2024-09-17T15:49:40.155 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T15:49:40.174 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:49:40.270 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: skanta@redhat.com first_in_suite: false job_id: '7909015' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC4xLWJ353EqAA8CBGLNR4urZmSC/KeKWZSkSfxUnZ55qfKt4C4BIIncLlbs6PBP4Lj5d1mrdMou59vTnPekfnU= smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIDjkEfvEI08ayJYXBLdWl/N3OpN6iqDJeeZ48TfT+0F3G4BtcU+fvbuXdIgO9NMoG5NEVNLDyiKykRZbTNAbU= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1xpK4yttHOVDgw5AKw8mOQPv4I4SciXhDvIuBWKq/jOGje+IqmqmDC3JlBNemPoCEecSDE+o6wPQ78lStIcl4= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWyqsqxRsfUAs1kNo0Fx71ryEubVxTm7cBWYH8rSWgaB+D6IZHUWWd5BRPx9NGo6Xrm8FCE95rJIf1CvF6VpnY= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: aggressive_pg_num_changes: false chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T15:49:40.270 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T15:49:40.271 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T15:49:40.271 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T15:49:40.272 INFO:teuthology.task.internal:Checking packages... 2024-09-17T15:49:40.293 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T15:49:40.293 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:49:40.293 INFO:teuthology.packaging:ref: None 2024-09-17T15:49:40.293 INFO:teuthology.packaging:tag: None 2024-09-17T15:49:40.293 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:49:40.293 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:49:40.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:49:40.433 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975.g1889a165 2024-09-17T15:49:40.435 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T15:49:40.442 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T15:49:40.442 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T15:49:40.449 INFO:teuthology.task.internal:Saving configuration 2024-09-17T15:49:40.464 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T15:49:40.489 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T15:49:40.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:44:32.333964', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC4xLWJ353EqAA8CBGLNR4urZmSC/KeKWZSkSfxUnZ55qfKt4C4BIIncLlbs6PBP4Lj5d1mrdMou59vTnPekfnU='} 2024-09-17T15:49:40.535 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:44:32.330229', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIDjkEfvEI08ayJYXBLdWl/N3OpN6iqDJeeZ48TfT+0F3G4BtcU+fvbuXdIgO9NMoG5NEVNLDyiKykRZbTNAbU='} 2024-09-17T15:49:40.556 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:44:32.332191', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1xpK4yttHOVDgw5AKw8mOQPv4I4SciXhDvIuBWKq/jOGje+IqmqmDC3JlBNemPoCEecSDE+o6wPQ78lStIcl4='} 2024-09-17T15:49:40.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:44:32.335480', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWyqsqxRsfUAs1kNo0Fx71ryEubVxTm7cBWYH8rSWgaB+D6IZHUWWd5BRPx9NGo6Xrm8FCE95rJIf1CvF6VpnY='} 2024-09-17T15:49:40.575 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T15:49:40.581 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-17T15:49:40.582 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-17T15:49:40.582 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-17T15:49:40.582 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-17T15:49:40.582 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T15:49:40.677 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc5be833be0>, signals=[15]) 2024-09-17T15:49:40.677 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T15:49:40.683 INFO:teuthology.task.internal:Opening connections... 2024-09-17T15:49:40.683 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:49:40.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:49:40.759 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T15:49:40.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:49:40.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T15:49:40.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:49:40.913 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T15:49:40.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:49:40.989 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T15:49:40.996 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-09-17T15:49:41.014 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-09-17T15:49:41.014 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-09-17T15:49:41.072 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:49:41.073 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:49:41.074 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-09-17T15:49:41.101 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-09-17T15:49:41.120 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-09-17T15:49:41.120 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-09-17T15:49:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-09-17T15:49:41.177 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:49:41.178 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:49:41.179 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-09-17T15:49:41.209 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-09-17T15:49:41.228 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-09-17T15:49:41.228 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:49:41.287 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:49:41.288 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:49:41.288 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:49:41.288 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:49:41.288 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:49:41.288 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-09-17T15:49:41.326 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-17T15:49:41.344 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-17T15:49:41.344 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-17T15:49:41.403 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T15:49:41.404 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T15:49:41.404 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T15:49:41.404 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-17T15:49:41.434 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T15:49:41.443 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T15:49:41.449 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T15:49:41.449 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:49:41.452 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:49:41.454 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:49:41.457 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:49:41.474 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T15:49:41.480 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T15:49:41.480 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:49:41.509 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:49:41.512 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:49:41.514 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:49:41.574 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T15:49:41.651 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T15:49:41.665 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T15:49:41.666 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T15:49:41.666 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:49:41.666 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:49:41.667 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:49:41.667 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:49:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:49:41.685 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-17T15:49:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:49:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:49:41.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:49:41.687 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-17T15:49:41.687 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-09-17T15:49:41.688 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-17T15:49:41.743 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:49:41.743 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-503.el9.x86_64 2024-09-17T15:49:41.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-09-17T15:49:41.745 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:49:41.745 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:49:41.746 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-503.el9.x86_64 2024-09-17T15:49:41.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-09-17T15:49:41.746 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-17T15:49:41.746 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-17T15:49:41.747 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-503.el9.x86_64 2024-09-17T15:49:41.747 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-503.el9.x86_64 2024-09-17T15:49:41.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-09-17T15:49:44.347 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.3 MB 00:00 2024-09-17T15:49:45.537 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-17T15:49:46.027 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.3 MB 00:02 2024-09-17T15:49:46.230 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.3 MB 00:02 2024-09-17T15:49:47.388 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-17T15:49:50.047 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-17T15:49:50.948 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-09-17T15:49:52.148 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.5 MB 00:00 2024-09-17T15:49:54.298 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-17T15:49:54.540 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-17T15:49:55.423 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T15:49:55.460 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T15:49:56.524 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-17T15:49:56.841 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 124 kB/s | 19 kB 00:00 2024-09-17T15:49:57.436 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-17T15:49:57.728 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-17T15:50:01.116 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-17T15:50:02.759 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T15:50:03.081 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:50:03.081 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:50:03.173 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:50:03.181 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:50:03.182 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-17T15:50:03.183 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:50:03.183 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 79 M 2024-09-17T15:50:03.183 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 126 M 2024-09-17T15:50:03.183 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:50:03.351 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T15:50:03.709 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-09-17T15:50:04.640 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:50:04.640 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:50:04.727 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:50:04.734 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:50:04.734 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-17T15:50:04.734 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:50:04.734 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-17T15:50:04.735 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:50:04.736 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-17T15:50:04.736 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-17T15:50:04.737 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:50:04.901 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 23 kB/s | 36 kB 00:01 2024-09-17T15:50:05.043 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 177 kB/s | 36 kB 00:00 2024-09-17T15:50:05.349 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:50:05.349 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:50:05.430 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:50:05.436 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-09-17T15:50:05.437 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:50:05.438 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 79 M 2024-09-17T15:50:05.438 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 126 M 2024-09-17T15:50:05.438 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:50:05.769 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 485 kB/s | 36 kB 00:00 2024-09-17T15:50:07.977 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 19 kB 00:00 2024-09-17T15:50:09.952 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.7 MB/s | 16 MB 00:04 2024-09-17T15:50:10.507 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-17T15:50:11.709 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.1 MB/s | 35 MB 00:08 2024-09-17T15:50:11.859 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 1.8 MB/s | 16 MB 00:08 2024-09-17T15:50:12.010 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-17T15:50:12.219 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.4 MB/s | 29 MB 00:06 2024-09-17T15:50:12.534 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.5 MB/s | 35 MB 00:07 2024-09-17T15:50:14.727 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-17T15:50:14.727 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:50:14.728 INFO:teuthology.orchestra.run.smithi006.stdout:Total 8.5 MB/s | 79 MB 00:09 2024-09-17T15:50:15.436 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:50:15.808 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:50:15.808 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:50:15.825 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.6 MB/s | 29 MB 00:10 2024-09-17T15:50:15.826 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:50:15.826 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.2 MB/s | 79 MB 00:12 2024-09-17T15:50:15.912 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:50:16.334 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.5 MB/s | 29 MB 00:11 2024-09-17T15:50:16.334 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:50:16.335 INFO:teuthology.orchestra.run.smithi181.stdout:Total 6.8 MB/s | 79 MB 00:11 2024-09-17T15:50:16.540 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:50:16.807 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:50:16.807 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:50:16.910 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:50:16.910 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:50:17.014 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:50:17.347 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:50:17.347 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:50:17.917 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T15:50:17.927 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T15:50:18.009 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:50:18.014 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-09-17T15:50:18.015 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:50:18.016 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 79 M 2024-09-17T15:50:18.016 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 126 M 2024-09-17T15:50:18.016 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:50:18.330 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:50:18.331 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:50:18.830 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 65 kB/s | 36 kB 00:00 2024-09-17T15:50:19.371 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:50:19.797 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-17T15:50:20.065 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:50:20.145 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:20.472 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-17T15:50:20.536 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:50:20.766 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:50:21.114 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-17T15:50:21.114 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:50:21.115 INFO:teuthology.orchestra.run.smithi169.stdout:Total 25 MB/s | 79 MB 00:03 2024-09-17T15:50:21.214 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:50:21.304 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:21.445 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:50:21.534 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:21.796 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:50:22.146 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:50:22.146 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:50:22.916 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:23.049 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:23.142 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:50:23.142 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:50:24.136 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:24.266 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:24.280 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:24.413 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:25.586 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:50:26.235 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:50:26.325 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:28.174 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:28.217 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:29.103 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:50:29.237 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:29.245 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:29.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:29.388 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:29.434 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:33.425 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:34.076 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:50:34.131 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:34.179 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:34.571 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:50:39.046 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:12.461 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:12.509 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:12.800 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:12.813 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:12.813 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:51:12.813 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:51:12.813 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:51:12.848 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:13.204 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:13.204 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:51:13.204 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:51:13.205 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:51:13.649 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:51:13.653 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:13.694 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:14.016 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:14.066 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:14.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:51:14.066 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:51:14.066 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.067 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.067 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.067 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:51:14.067 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:51:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:51:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:51:14.076 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:51:14.100 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.102 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.102 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:51:14.102 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:51:14.102 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-17T15:51:14.102 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:51:14.102 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:14.161 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.161 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-17T15:51:14.200 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:51:14.201 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:51:14.202 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:51:14.203 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:51:14.203 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.204 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:51:14.418 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:14.513 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.514 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.514 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.514 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:51:14.514 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:51:14.514 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T15:51:14.514 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:51:14.514 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:14.574 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.574 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-09-17T15:51:14.617 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-09-17T15:51:14.617 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-09-17T15:51:14.617 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 82.el9 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8274670 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:51:14.618 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:51:14.619 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:51:14.620 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:51:14.620 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:51:14.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:51:14.839 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-17T15:51:14.896 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:14.896 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:51:14.896 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:51:14.897 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:51:15.220 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:15.280 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-09-17T15:51:15.324 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:15.325 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:15.325 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:51:15.325 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:51:15.325 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:51:15.325 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T15:51:15.325 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:51:15.325 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:15.384 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:15.385 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:15.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 82.el9 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8274670 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-09-17T15:51:15.417 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:51:15.418 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-09-17T15:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:51:15.419 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:51:15.420 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:51:15.420 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:51:15.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:51:16.039 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-09-17T15:51:16.138 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:51:16.173 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-17T15:51:16.176 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-17T15:51:16.193 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.1u2dux4D1b 2024-09-17T15:51:16.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1u2dux4D1b 2024-09-17T15:51:16.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.1u2dux4D1b 2024-09-17T15:51:16.400 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.1u2dux4D1b is 6KB 2024-09-17T15:51:16.448 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.1u2dux4D1b 2024-09-17T15:51:16.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:51:16.525 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:51:16.526 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:51:16.526 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:51:16.526 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:51:16.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:51:16.649 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-17T15:51:16.688 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:51:16.726 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-09-17T15:51:16.728 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-17T15:51:16.743 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.ln7fkkm5UZ 2024-09-17T15:51:16.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ln7fkkm5UZ 2024-09-17T15:51:16.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.ln7fkkm5UZ 2024-09-17T15:51:16.950 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.ln7fkkm5UZ is 6KB 2024-09-17T15:51:16.970 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.ln7fkkm5UZ 2024-09-17T15:51:16.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:51:17.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:51:17.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-09-17T15:51:17.319 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:17.358 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:17.366 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:51:17.404 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-09-17T15:51:17.406 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-17T15:51:17.449 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.iOjUqokd0L 2024-09-17T15:51:17.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iOjUqokd0L 2024-09-17T15:51:17.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.iOjUqokd0L 2024-09-17T15:51:17.655 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.iOjUqokd0L is 6KB 2024-09-17T15:51:17.704 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.iOjUqokd0L 2024-09-17T15:51:17.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:51:17.725 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:17.725 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T15:51:17.726 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T15:51:17.726 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T15:51:17.781 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:51:17.782 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:51:17.782 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:51:17.782 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:51:17.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:51:17.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-09-17T15:51:18.598 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T15:51:18.598 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:51:18.598 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:51:18.598 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:18.599 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:18.599 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T15:51:18.599 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T15:51:18.599 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:51:18.599 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:51:18.952 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:19.034 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:19.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:19.035 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T15:51:19.035 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:51:19.035 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:51:19.035 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-17T15:51:19.035 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:51:19.035 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T15:51:19.095 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T15:51:19.095 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T15:51:19.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-09-17T15:51:19.151 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-09-17T15:51:19.151 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-09-17T15:51:19.151 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-09-17T15:51:19.151 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 82.el9 2024-09-17T15:51:19.151 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8274670 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-09-17T15:51:19.152 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:51:19.153 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T15:51:19.154 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T15:51:19.154 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-508.el9.x86_64 2024-09-17T15:51:19.154 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T15:51:19.734 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-09-17T15:51:21.000 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T15:51:21.025 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-09-17T15:51:21.027 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-09-17T15:51:21.060 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.1qiEyZWKxE 2024-09-17T15:51:21.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1qiEyZWKxE 2024-09-17T15:51:21.128 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.1qiEyZWKxE 2024-09-17T15:51:21.271 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.1qiEyZWKxE is 6KB 2024-09-17T15:51:21.319 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.1qiEyZWKxE 2024-09-17T15:51:21.334 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T15:51:21.405 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T15:51:21.405 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T15:51:21.405 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T15:51:21.405 INFO:teuthology.orchestra.run.smithi169.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T15:51:21.407 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T15:51:21.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-09-17T15:51:46.658 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-17T15:51:46.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:51:46.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:51:47.215 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T15:51:47.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:51:47.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:51:47.931 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T15:51:47.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:51:47.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:51:51.562 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-17T15:51:51.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:51:51.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:05.181 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-17T15:52:05.693 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-09-17T15:52:06.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-09-17T15:52:10.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-17T15:52:14.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:52:14.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:14.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:52:14.695 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:15.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:52:15.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:17.245 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.181 2024-09-17T15:52:18.525 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-09-17T15:52:19.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:52:19.047 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:22.109 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-17T15:52:29.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:52:29.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:30.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:52:30.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:34.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:52:34.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:37.181 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-17T15:52:52.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:52:52.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:52:55.261 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-09-17T15:53:01.094 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-17T15:53:01.496 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:53:01.496 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:53:01.496 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-17T15:53:01.511 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:53:01.511 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:53:01.511 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:53:01.511 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:53:02.302 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-17T15:53:02.512 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-17T15:53:02.512 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:53:02.588 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-17T15:53:02.616 DEBUG:teuthology.parallel:result is None 2024-09-17T15:53:02.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:53:02.804 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:53:02.804 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-17T15:53:02.820 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:53:02.820 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:53:02.820 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:53:02.820 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:53:03.821 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T15:53:03.821 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:53:03.882 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-09-17T15:53:03.907 DEBUG:teuthology.parallel:result is None 2024-09-17T15:53:13.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:53:13.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:53:13.641 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-09-17T15:53:13.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:53:13.970 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:53:13.970 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-09-17T15:53:13.986 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:53:13.986 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:53:13.986 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:53:13.986 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:53:14.698 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T15:53:14.987 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-17T15:53:14.987 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:53:15.062 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-09-17T15:53:15.080 DEBUG:teuthology.parallel:result is None 2024-09-17T15:53:26.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:53:26.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:53:27.014 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-17T15:53:27.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:53:27.411 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T15:53:27.411 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-09-17T15:53:27.428 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-508.el9.x86_64 2024-09-17T15:53:27.428 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T15:53:27.428 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:53:27.428 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T15:53:28.429 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T15:53:28.429 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:53:28.520 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-09-17T15:53:28.541 DEBUG:teuthology.parallel:result is None 2024-09-17T15:53:28.541 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T15:53:28.549 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T15:53:28.549 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:53:28.552 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:53:28.554 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:53:28.557 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:53:28.576 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T15:53:28.586 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T15:53:28.600 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T15:53:28.600 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:28.611 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:28.614 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:28.616 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:28.698 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T15:53:28.709 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T15:53:28.709 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:53:28.711 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:53:28.714 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:53:28.716 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:53:28.746 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.749 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.753 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.755 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.758 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.762 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.765 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.768 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:53:28.770 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T15:53:28.781 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T15:53:28.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:53:28.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:53:28.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:53:28.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:53:28.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T15:53:28.850 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T15:53:28.850 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:53:28.876 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:53:28.879 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:53:28.882 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:53:28.898 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:28.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.029 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.160 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:53:29.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:53:29.229 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.327 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.458 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:53:29.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:53:29.528 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.635 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.678 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:29.722 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:53:29.723 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:53:29.789 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:29.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:53:30.081 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:30.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:53:30.171 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:53:30.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:53:30.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-09-17T15:53:30.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-09-17T15:53:30.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-09-17T15:53:30.247 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-17T15:53:30.284 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:53:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:53:30.297 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:53:30.336 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T15:53:30.806 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T15:53:30.813 INFO:teuthology.task.internal:Starting timer... 2024-09-17T15:53:30.814 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T15:53:30.837 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T15:53:30.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-09-17T15:53:30.921 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-09-17T15:53:30.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-09-17T15:53:30.965 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-09-17T15:53:30.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-09-17T15:53:31.020 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-09-17T15:53:31.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-17T15:53:31.085 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-17T15:53:31.087 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T15:53:31.087 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-09-17T15:53:31.117 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-09-17T15:53:31.117 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-09-17T15:53:31.145 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-09-17T15:53:31.146 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-09-17T15:53:31.169 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-09-17T15:53:31.170 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-17T15:53:31.197 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-17T15:53:31.197 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi050.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-17T15:53:31.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:53:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:31.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:53:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:31.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:53:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:31.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T15:53:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:31.321 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T15:53:31.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:53:31.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:53:31.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:53:31.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-17T15:53:31.471 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T15:53:31.480 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T15:53:31.492 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T15:53:31.494 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi169.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T15:57:22.541 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-17T15:57:22.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:57:22.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:57:22.621 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-09-17T15:57:22.705 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-09-17T15:57:22.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:57:22.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:57:22.783 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-17T15:57:22.865 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-17T15:57:22.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:57:22.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:57:22.943 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-09-17T15:57:23.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-09-17T15:57:23.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:57:23.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:57:23.103 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-17T15:57:23.183 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T15:57:23.184 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T15:57:23.194 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T15:57:23.195 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:57:23.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:57:23.198 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:57:23.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:57:23.201 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:57:23.201 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:57:23.203 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:57:23.203 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:57:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:57:23.230 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:57:23.233 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:57:23.245 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:57:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:57:23.250 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:57:23.258 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T15:57:23.270 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-09-17T15:57:23.274 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T15:57:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-09-17T15:57:23.278 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-09-17T15:57:23.281 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-09-17T15:57:23.287 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-09-17T15:57:23.290 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-09-17T15:57:23.295 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:57:23.301 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-17T15:57:23.303 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:57:23.306 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:57:23.309 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:57:23.312 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-17T15:57:23.318 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:57:23.322 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:57:23.327 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T15:57:23.342 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T15:57:23.366 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.369 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.376 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.380 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.385 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:57:23.389 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:57:23.389 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-17T15:57:23.389 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.389 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.390 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.390 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.396 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.400 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T15:57:23.401 INFO:teuthology.run_tasks:Running task install... 2024-09-17T15:57:23.409 DEBUG:teuthology.task.install:project ceph 2024-09-17T15:57:23.410 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:57:23.410 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T15:57:23.410 INFO:teuthology.task.install:Using flavor: default 2024-09-17T15:57:23.415 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T15:57:23.416 INFO:teuthology.task.install:extra packages: [] 2024-09-17T15:57:23.416 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:57:23.417 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:57:23.419 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:57:23.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:57:23.420 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:57:23.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:57:23.421 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'tag': None, 'wait_for_package': False} 2024-09-17T15:57:23.421 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:57:23.566 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:57:23.566 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:57:23.584 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:57:23.584 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:57:23.587 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:57:23.587 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:57:23.599 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/ 2024-09-17T15:57:23.599 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4975.g1889a165 2024-09-17T15:57:23.720 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:57:23.720 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:57:23.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:57:23.748 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:57:23.748 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:57:23.767 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:57:23.767 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:57:23.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:57:23.771 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:57:23.771 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:57:23.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:57:23.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:57:23.797 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:57:23.798 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:57:23.799 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:57:23.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:57:23.864 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:57:23.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:57:23.893 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:57:23.939 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:57:23.942 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:57:23.958 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-09-17T15:57:23.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-09-17T15:57:24.004 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-09-17T15:57:24.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-09-17T15:57:24.007 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-09-17T15:57:24.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-09-17T15:57:24.195 INFO:teuthology.orchestra.run.smithi006.stdout:45 files removed 2024-09-17T15:57:24.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2024-09-17T15:57:24.247 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-09-17T15:57:24.250 INFO:teuthology.orchestra.run.smithi169.stdout:45 files removed 2024-09-17T15:57:24.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-09-17T15:57:24.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-09-17T15:57:24.861 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 247 kB/s | 84 kB 00:00 2024-09-17T15:57:24.893 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 254 kB/s | 84 kB 00:00 2024-09-17T15:57:24.903 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 255 kB/s | 84 kB 00:00 2024-09-17T15:57:25.154 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-17T15:57:25.193 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-17T15:57:25.195 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-17T15:57:25.432 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T15:57:25.471 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T15:57:25.477 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-17T15:57:25.936 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.3 MB 00:00 2024-09-17T15:57:26.087 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-17T15:57:26.729 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T15:57:26.730 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:57:26.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T15:57:26.761 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-17T15:57:26.761 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1889a165b5128787ec693450a3d1c5df5d6dc203/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T15:57:26.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T15:57:26.908 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T15:57:26.974 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-17T15:57:26.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-17T15:57:27.115 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.3 MB 00:01 2024-09-17T15:57:27.204 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-17T15:57:27.232 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-17T15:57:27.788 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-17T15:57:27.830 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 32 MB/s | 20 MB 00:00 2024-09-17T15:57:28.080 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-17T15:57:28.356 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 8.0 kB/s | 1.9 kB 00:00 2024-09-17T15:57:28.562 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-09-17T15:57:28.908 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-17T15:57:29.214 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.3 MB 00:00 2024-09-17T15:57:31.116 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-17T15:57:32.407 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-17T15:57:33.957 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-17T15:57:34.363 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.5 MB 00:01 2024-09-17T15:57:34.542 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-17T15:57:34.778 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-17T15:57:36.285 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 19 kB 00:00 2024-09-17T15:57:36.938 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-17T15:57:37.408 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.5 MB 00:02 2024-09-17T15:57:37.439 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-17T15:57:37.709 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-17T15:57:38.875 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 19 kB 00:00 2024-09-17T15:57:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-17T15:57:40.492 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T15:57:42.796 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:57:42.804 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:57:42.805 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:57:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout:Installing weak dependencies: 2024-09-17T15:57:42.807 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:Install 31 Packages 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 62 M 2024-09-17T15:57:42.808 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:57:43.103 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T15:57:43.107 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-17T15:57:43.310 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-17T15:57:43.361 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 503 kB/s | 25 kB 00:00 2024-09-17T15:57:43.444 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 8.9 MB/s | 756 kB 00:00 2024-09-17T15:57:43.536 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 5.4 MB/s | 508 kB 00:00 2024-09-17T15:57:43.886 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): librgw2-19.3.0-4975.g1889a165.el9.x86_6 16 MB/s | 5.5 MB 00:00 2024-09-17T15:57:44.012 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 9.7 MB/s | 11 MB 00:01 2024-09-17T15:57:44.037 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-ceph-argparse-19.3.0-4975.g1889 300 kB/s | 45 kB 00:00 2024-09-17T15:57:44.062 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-ceph-common-19.3.0-4975.g1889a1 2.8 MB/s | 143 kB 00:00 2024-09-17T15:57:44.087 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 3.1 MB/s | 161 kB 00:00 2024-09-17T15:57:44.121 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 5.3 MB/s | 321 kB 00:00 2024-09-17T15:57:44.146 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 5.1 MB/s | 305 kB 00:00 2024-09-17T15:57:44.172 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T15:57:44.389 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): ceph-common-19.3.0-4975.g1889a165.el9. 14 MB/s | 22 MB 00:01 2024-09-17T15:57:44.456 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-17T15:57:44.506 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 511 kB/s | 171 kB 00:00 2024-09-17T15:57:44.531 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-17T15:57:44.556 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 959 kB/s | 160 kB 00:00 2024-09-17T15:57:44.590 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-09-17T15:57:44.615 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-09-17T15:57:44.640 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-17T15:57:44.666 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-17T15:57:44.691 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-09-17T15:57:44.716 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-17T15:57:44.775 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T15:57:44.800 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T15:57:44.825 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 280 kB/s | 26 kB 00:00 2024-09-17T15:57:44.851 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T15:57:44.876 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 7.5 MB/s | 191 kB 00:00 2024-09-17T15:57:44.926 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-17T15:57:44.968 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-17T15:57:45.052 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T15:57:45.177 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 16 MB/s | 3.3 MB 00:00 2024-09-17T15:57:45.236 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): librados2-19.3.0-4975.g1889a165.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-17T15:57:45.241 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:57:45.241 INFO:teuthology.orchestra.run.smithi050.stdout:Total 25 MB/s | 62 MB 00:02 2024-09-17T15:57:45.359 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:57:45.366 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:57:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:57:45.372 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout:Install 31 Packages 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:57:45.373 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 62 M 2024-09-17T15:57:45.374 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:57:45.400 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:57:45.400 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:57:45.459 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:57:45.473 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:57:45.474 INFO:teuthology.orchestra.run.smithi006.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:45.475 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:57:45.476 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:57:45.477 INFO:teuthology.orchestra.run.smithi006.stdout:Install 31 Packages 2024-09-17T15:57:45.478 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2024-09-17T15:57:45.478 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:57:45.479 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 62 M 2024-09-17T15:57:45.479 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:57:45.494 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-17T15:57:46.056 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:57:46.057 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:57:46.105 INFO:teuthology.orchestra.run.smithi006.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-17T15:57:46.172 INFO:teuthology.orchestra.run.smithi006.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 378 kB/s | 25 kB 00:00 2024-09-17T15:57:46.322 INFO:teuthology.orchestra.run.smithi006.stdout:(3/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 14 MB/s | 11 MB 00:00 2024-09-17T15:57:46.356 INFO:teuthology.orchestra.run.smithi006.stdout:(4/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 4.0 MB/s | 756 kB 00:00 2024-09-17T15:57:46.389 INFO:teuthology.orchestra.run.smithi006.stdout:(5/33): libradosstriper1-19.3.0-4975.g1889a165. 7.4 MB/s | 508 kB 00:00 2024-09-17T15:57:46.498 INFO:teuthology.orchestra.run.smithi006.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 417 kB/s | 45 kB 00:00 2024-09-17T15:57:46.598 INFO:teuthology.orchestra.run.smithi006.stdout:(7/33): python3-ceph-common-19.3.0-4975.g1889a1 1.4 MB/s | 143 kB 00:00 2024-09-17T15:57:46.682 INFO:teuthology.orchestra.run.smithi006.stdout:(8/33): python3-cephfs-19.3.0-4975.g1889a165.el 1.9 MB/s | 161 kB 00:00 2024-09-17T15:57:46.782 INFO:teuthology.orchestra.run.smithi006.stdout:(9/33): python3-rados-19.3.0-4975.g1889a165.el9 3.1 MB/s | 321 kB 00:00 2024-09-17T15:57:46.857 INFO:teuthology.orchestra.run.smithi006.stdout:(10/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-17T15:57:46.866 INFO:teuthology.orchestra.run.smithi169.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 6.3 MB/s | 5.5 MB 00:00 2024-09-17T15:57:46.884 INFO:teuthology.orchestra.run.smithi006.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 3.0 MB/s | 305 kB 00:00 2024-09-17T15:57:46.916 INFO:teuthology.orchestra.run.smithi006.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.7 MB/s | 99 kB 00:00 2024-09-17T15:57:46.933 INFO:teuthology.orchestra.run.smithi169.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 378 kB/s | 25 kB 00:00 2024-09-17T15:57:47.034 INFO:teuthology.orchestra.run.smithi169.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 7.4 MB/s | 756 kB 00:00 2024-09-17T15:57:47.109 INFO:teuthology.orchestra.run.smithi169.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 6.6 MB/s | 508 kB 00:00 2024-09-17T15:57:47.150 INFO:teuthology.orchestra.run.smithi006.stdout:(13/33): ceph-common-19.3.0-4975.g1889a165.el9. 14 MB/s | 22 MB 00:01 2024-09-17T15:57:47.192 INFO:teuthology.orchestra.run.smithi006.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-09-17T15:57:47.234 INFO:teuthology.orchestra.run.smithi006.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T15:57:47.251 INFO:teuthology.orchestra.run.smithi169.stdout:(5/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 8.6 MB/s | 11 MB 00:01 2024-09-17T15:57:47.259 INFO:teuthology.orchestra.run.smithi006.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 498 kB/s | 171 kB 00:00 2024-09-17T15:57:47.292 INFO:teuthology.orchestra.run.smithi006.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-17T15:57:47.309 INFO:teuthology.orchestra.run.smithi169.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 778 kB/s | 45 kB 00:00 2024-09-17T15:57:47.326 INFO:teuthology.orchestra.run.smithi006.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 662 kB 00:00 2024-09-17T15:57:47.351 INFO:teuthology.orchestra.run.smithi006.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 719 kB/s | 42 kB 00:00 2024-09-17T15:57:47.377 INFO:teuthology.orchestra.run.smithi006.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-17T15:57:47.384 INFO:teuthology.orchestra.run.smithi169.stdout:(7/33): python3-ceph-common-19.3.0-4975.g1889a1 1.9 MB/s | 143 kB 00:00 2024-09-17T15:57:47.402 INFO:teuthology.orchestra.run.smithi006.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 717 kB/s | 36 kB 00:00 2024-09-17T15:57:47.427 INFO:teuthology.orchestra.run.smithi006.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-17T15:57:47.443 INFO:teuthology.orchestra.run.smithi169.stdout:(8/33): python3-cephfs-19.3.0-4975.g1889a165.el 2.7 MB/s | 161 kB 00:00 2024-09-17T15:57:47.452 INFO:teuthology.orchestra.run.smithi006.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-17T15:57:47.510 INFO:teuthology.orchestra.run.smithi169.stdout:(9/33): python3-rados-19.3.0-4975.g1889a165.el9 4.7 MB/s | 321 kB 00:00 2024-09-17T15:57:47.511 INFO:teuthology.orchestra.run.smithi006.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T15:57:47.537 INFO:teuthology.orchestra.run.smithi006.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T15:57:47.562 INFO:teuthology.orchestra.run.smithi006.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 279 kB/s | 26 kB 00:00 2024-09-17T15:57:47.585 INFO:teuthology.orchestra.run.smithi169.stdout:(10/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 5.5 MB 00:00 2024-09-17T15:57:47.587 INFO:teuthology.orchestra.run.smithi006.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T15:57:47.621 INFO:teuthology.orchestra.run.smithi006.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:57:47.641 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:57:47.663 INFO:teuthology.orchestra.run.smithi006.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-17T15:57:47.711 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:57:47.725 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:57:47.730 INFO:teuthology.orchestra.run.smithi006.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-17T15:57:47.745 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:57:47.745 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:57:47.745 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:57:47.745 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 11 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.5 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.3 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 22 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-4975.g1889a165.el9 ceph 25 k 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T15:57:47.746 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 756 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 508 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-4975.g1889a165.el9 ceph 5.5 M 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 45 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-4975.g1889a165.el9 ceph 143 k 2024-09-17T15:57:47.747 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-4975.g1889a165.el9 ceph 161 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-4975.g1889a165.el9 ceph 321 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 305 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-4975.g1889a165.el9 ceph 99 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:57:47.748 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout:Install 31 Packages 2024-09-17T15:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-17T15:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 62 M 2024-09-17T15:57:47.749 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:57:47.758 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:57:47.783 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:57:47.802 INFO:teuthology.orchestra.run.smithi169.stdout:(11/33): ceph-common-19.3.0-4975.g1889a165.el9. 12 MB/s | 22 MB 00:01 2024-09-17T15:57:47.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:47.813 INFO:teuthology.orchestra.run.smithi006.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-17T15:57:47.819 INFO:teuthology.orchestra.run.smithi169.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 425 kB/s | 99 kB 00:00 2024-09-17T15:57:47.833 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:47.844 INFO:teuthology.orchestra.run.smithi169.stdout:(13/33): python3-rbd-19.3.0-4975.g1889a165.el9. 914 kB/s | 305 kB 00:00 2024-09-17T15:57:47.961 INFO:teuthology.orchestra.run.smithi169.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-17T15:57:47.988 INFO:teuthology.orchestra.run.smithi169.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2024-09-17T15:57:48.005 INFO:teuthology.orchestra.run.smithi006.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 10 MB/s | 3.5 MB 00:00 2024-09-17T15:57:48.012 INFO:teuthology.orchestra.run.smithi169.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 959 kB/s | 160 kB 00:00 2024-09-17T15:57:48.026 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:48.029 INFO:teuthology.orchestra.run.smithi169.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 675 kB/s | 45 kB 00:00 2024-09-17T15:57:48.064 INFO:teuthology.orchestra.run.smithi006.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 9.7 MB/s | 3.3 MB 00:00 2024-09-17T15:57:48.069 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:57:48.069 INFO:teuthology.orchestra.run.smithi006.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-17T15:57:48.070 INFO:teuthology.orchestra.run.smithi169.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-17T15:57:48.086 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:48.113 INFO:teuthology.orchestra.run.smithi169.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-09-17T15:57:48.146 INFO:teuthology.orchestra.run.smithi169.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-17T15:57:48.163 INFO:teuthology.orchestra.run.smithi169.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-17T15:57:48.188 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:57:48.188 INFO:teuthology.orchestra.run.smithi169.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-17T15:57:48.214 INFO:teuthology.orchestra.run.smithi169.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 705 kB/s | 47 kB 00:00 2024-09-17T15:57:48.229 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:57:48.229 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:57:48.273 INFO:teuthology.orchestra.run.smithi169.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T15:57:48.289 INFO:teuthology.orchestra.run.smithi169.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 3.3 MB/s | 49 kB 00:00 2024-09-17T15:57:48.306 INFO:teuthology.orchestra.run.smithi169.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 337 kB/s | 26 kB 00:00 2024-09-17T15:57:48.332 INFO:teuthology.orchestra.run.smithi169.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-17T15:57:48.365 INFO:teuthology.orchestra.run.smithi169.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:57:48.441 INFO:teuthology.orchestra.run.smithi169.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.1 MB/s | 839 kB 00:00 2024-09-17T15:57:48.441 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:57:48.483 INFO:teuthology.orchestra.run.smithi169.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-17T15:57:48.558 INFO:teuthology.orchestra.run.smithi169.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T15:57:48.559 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:48.674 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:48.768 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:57:48.794 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:48.800 INFO:teuthology.orchestra.run.smithi169.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 9.7 MB/s | 3.5 MB 00:00 2024-09-17T15:57:48.849 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:48.850 INFO:teuthology.orchestra.run.smithi169.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 8.9 MB/s | 3.3 MB 00:00 2024-09-17T15:57:48.855 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:57:48.855 INFO:teuthology.orchestra.run.smithi169.stdout:Total 18 MB/s | 62 MB 00:03 2024-09-17T15:57:48.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:57:48.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:57:48.949 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:57:48.950 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:57:48.969 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:57:49.000 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:57:49.006 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:57:49.006 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:57:49.029 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:49.062 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-base-19.3.0-4975.g1889a165.el9.x86 8.0 MB/s | 5.5 MB 00:00 2024-09-17T15:57:49.085 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:49.104 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-selinux-19.3.0-4975.g1889a165.el9. 608 kB/s | 25 kB 00:00 2024-09-17T15:57:49.119 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:57:49.230 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:57:49.237 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): libcephfs2-19.3.0-4975.g1889a165.el9.x8 5.5 MB/s | 756 kB 00:00 2024-09-17T15:57:49.270 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:57:49.311 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:57:49.337 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): libradosstriper1-19.3.0-4975.g1889a165. 5.0 MB/s | 508 kB 00:00 2024-09-17T15:57:49.344 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:57:49.373 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:57:49.412 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:57:49.452 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:57:49.463 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): ceph-radosgw-19.3.0-4975.g1889a165.el9. 10 MB/s | 11 MB 00:01 2024-09-17T15:57:49.530 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): python3-ceph-argparse-19.3.0-4975.g1889 678 kB/s | 45 kB 00:00 2024-09-17T15:57:49.677 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:57:49.677 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:57:49.756 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): ceph-common-19.3.0-4975.g1889a165.el9.x 16 MB/s | 22 MB 00:01 2024-09-17T15:57:49.781 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-ceph-common-19.3.0-4975.g1889a1 572 kB/s | 143 kB 00:00 2024-09-17T15:57:49.788 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:57:49.806 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): python3-cephfs-19.3.0-4975.g1889a165.el 3.1 MB/s | 161 kB 00:00 2024-09-17T15:57:49.827 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:49.831 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): python3-rados-19.3.0-4975.g1889a165.el 6.3 MB/s | 321 kB 00:00 2024-09-17T15:57:49.857 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-rbd-19.3.0-4975.g1889a165.el9. 6.0 MB/s | 305 kB 00:00 2024-09-17T15:57:49.881 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-rgw-19.3.0-4975.g1889a165.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T15:57:49.920 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:49.949 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:57:49.982 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): librgw2-19.3.0-4975.g1889a165.el9.x86_ 8.5 MB/s | 5.5 MB 00:00 2024-09-17T15:57:49.985 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:57:50.032 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 609 kB/s | 107 kB 00:00 2024-09-17T15:57:50.082 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 910 kB/s | 45 kB 00:00 2024-09-17T15:57:50.116 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 729 kB/s | 171 kB 00:00 2024-09-17T15:57:50.141 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-17T15:57:50.183 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-17T15:57:50.208 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 631 kB/s | 42 kB 00:00 2024-09-17T15:57:50.234 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-17T15:57:50.259 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 715 kB/s | 36 kB 00:00 2024-09-17T15:57:50.284 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-09-17T15:57:50.310 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-17T15:57:50.369 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-17T15:57:50.393 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T15:57:50.419 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 277 kB/s | 26 kB 00:00 2024-09-17T15:57:50.444 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-17T15:57:50.478 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-17T15:57:50.492 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:57:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:57:50.528 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-17T15:57:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:50.578 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-17T15:57:50.581 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:57:50.602 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:50.610 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:57:50.636 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:57:50.658 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:50.662 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-17T15:57:50.683 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:57:50.687 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:50.715 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:57:50.837 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): librados2-19.3.0-4975.g1889a165.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-17T15:57:50.887 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:50.904 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): librbd1-19.3.0-4975.g1889a165.el9.x86_ 10 MB/s | 3.3 MB 00:00 2024-09-17T15:57:50.909 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:57:50.909 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 62 MB 00:03 2024-09-17T15:57:50.939 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:51.016 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:57:51.052 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:57:51.052 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:57:51.196 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:57:51.271 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:57:51.295 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:57:51.305 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:57:51.339 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:57:51.361 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:51.389 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:51.413 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:51.506 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:51.581 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:51.613 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:57:51.642 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:51.647 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:51.694 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:51.722 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:57:51.737 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:57:51.737 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:57:51.766 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:57:51.837 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:57:51.866 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:51.931 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:51.981 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:57:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:57:52.101 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:57:52.124 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:52.133 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:57:52.173 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:57:52.214 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:57:52.217 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:52.244 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:57:52.277 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:57:52.314 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:57:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:57:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:52.414 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:57:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:57:52.563 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:57:52.586 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:52.650 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:52.657 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:57:52.685 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:57:52.698 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:52.774 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:57:52.803 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:57:52.836 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:57:52.839 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:57:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:57:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:57:52.955 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:57:52.989 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:57:53.029 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:57:53.189 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:53.220 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:53.335 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:57:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:57:53.358 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:57:53.384 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:53.413 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-17T15:57:53.451 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-17T15:57:53.456 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-17T15:57:53.499 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:53.508 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:53.521 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-17T15:57:53.540 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:57:53.576 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:57:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:57:53.694 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:57:53.710 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:53.764 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-17T15:57:53.823 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:53.873 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:53.967 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:53.968 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:57:53.968 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:57:53.968 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:57:54.008 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:57:54.066 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:57:54.104 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:54.159 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-17T15:57:54.175 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:54.245 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:54.245 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:57:54.281 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:57:54.337 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:57:54.430 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 8/35 2024-09-17T15:57:54.455 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:54.517 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 9/35 2024-09-17T15:57:54.546 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-17T15:57:54.590 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 11/35 2024-09-17T15:57:54.660 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:57:54.682 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:54.745 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 13/35 2024-09-17T15:57:54.780 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-17T15:57:54.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-17T15:57:54.931 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-17T15:57:54.978 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-17T15:57:55.013 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-17T15:57:55.042 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-17T15:57:55.075 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:57:55.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-17T15:57:55.429 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-17T15:57:55.472 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:55.563 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 23/35 2024-09-17T15:57:55.593 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 24/35 2024-09-17T15:57:55.637 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-17T15:57:56.138 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-17T15:57:56.164 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:56.166 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:56.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:56.237 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 27/35 2024-09-17T15:57:56.316 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 28/35 2024-09-17T15:57:56.352 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 29/35 2024-09-17T15:57:56.579 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:56.611 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:56.803 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:56.845 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:56.941 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:56.941 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:57:56.941 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:57:56.941 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:57:56.979 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:57:57.179 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:57.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:57.297 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:57.297 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:57:57.297 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:57:57.297 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:57:57.324 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:57:58.647 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:58.815 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:59.385 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 30/35 2024-09-17T15:57:59.426 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 31/35 2024-09-17T15:57:59.517 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T15:57:59.517 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T15:57:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:57:59.561 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:04.701 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:04.701 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-09-17T15:58:04.701 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-09-17T15:58:04.701 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-09-17T15:58:04.702 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-09-17T15:58:04.702 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-09-17T15:58:04.702 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-09-17T15:58:04.702 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-09-17T15:58:04.702 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:05.735 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:05.762 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:05.762 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:05.762 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:58:05.762 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:05.763 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:05.763 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:05.764 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:05.862 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:58:07.099 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:58:07.100 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:58:07.101 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:58:07.102 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-09-17T15:58:07.379 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-09-17T15:58:07.380 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-09-17T15:58:07.380 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-09-17T15:58:07.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-09-17T15:58:07.384 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.154 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:58:08.155 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:58:08.156 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:58:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:58:08.157 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:08.157 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:08.413 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:08.426 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:08.440 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:08.440 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:08.440 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:58:08.440 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:08.440 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:08.441 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:08.442 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:08.452 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:08.454 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:08.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-09-17T15:58:08.532 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:08.532 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:08.536 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:08.537 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:09.070 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:09.214 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:09.214 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:09.214 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-17T15:58:09.215 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:09.216 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 46 M 2024-09-17T15:58:09.216 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 201 M 2024-09-17T15:58:09.216 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:09.717 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 732 kB/s | 162 kB 00:00 2024-09-17T15:58:09.721 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:58:09.722 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:58:09.723 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:58:09.724 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:58:09.742 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-17T15:58:09.852 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:09.852 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:58:09.852 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:58:09.853 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:58:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:58:09.855 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:58:09.855 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:58:09.855 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:58:09.855 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:58:09.855 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 477 kB/s | 64 kB 00:00 2024-09-17T15:58:09.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:58:09.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:58:09.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:09.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:58:09.856 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-17T15:58:09.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.541 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.542 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.543 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:10.544 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:10.892 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:58:10.893 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:58:10.894 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:10.895 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:10.914 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-09-17T15:58:10.962 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:10.988 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 33/35 2024-09-17T15:58:10.988 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:10.989 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T15:58:10.989 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:10.989 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:58:10.989 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:10.990 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-17T15:58:11.082 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:11.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2024-09-17T15:58:11.451 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 23 MB/s | 46 MB 00:01 2024-09-17T15:58:11.453 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:11.454 INFO:teuthology.orchestra.run.smithi050.stdout:Total 21 MB/s | 46 MB 00:02 2024-09-17T15:58:11.483 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:11.494 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:11.506 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:11.506 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:11.625 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:11.625 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:11.625 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-09-17T15:58:11.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:11.627 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 46 M 2024-09-17T15:58:11.627 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 201 M 2024-09-17T15:58:11.627 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:11.838 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:28 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:11.897 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:11.897 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:11.974 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:58:11.975 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 46 M 2024-09-17T15:58:11.976 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 201 M 2024-09-17T15:58:11.977 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:12.067 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-17T15:58:12.110 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T15:58:12.135 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 705 kB/s | 162 kB 00:00 2024-09-17T15:58:12.171 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-17T15:58:12.213 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 1/35 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2/35 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 3/35 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 4/35 2024-09-17T15:58:12.291 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 5/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_6 6/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 7/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9. 8/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x8 9/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 10/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 11/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 12/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 13/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-17T15:58:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-17T15:58:12.293 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-4975.g1889a165.el9.x86_64 32/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-17T15:58:12.294 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 34/35 2024-09-17T15:58:12.296 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 593 kB/s | 162 kB 00:00 2024-09-17T15:58:12.494 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:12.586 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:58:12.629 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:58:12.653 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:13.306 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-17T15:58:13.306 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:13.307 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-17T15:58:13.307 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T15:58:13.308 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T15:58:13.309 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:13.310 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:13.419 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 30 MB/s | 46 MB 00:01 2024-09-17T15:58:13.421 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:13.422 INFO:teuthology.orchestra.run.smithi169.stdout:Total 26 MB/s | 46 MB 00:01 2024-09-17T15:58:13.450 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:13.471 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:13.471 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:13.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-17T15:58:13.840 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:13.841 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:14.256 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:29 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:14.358 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:14.389 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:14.389 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:14.389 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:58:14.389 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-4975.g1889a165.el9 ceph 46 M 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-4975.g1889a165.el9 ceph 162 k 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-17T15:58:14.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:14.391 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 46 M 2024-09-17T15:58:14.391 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 201 M 2024-09-17T15:58:14.391 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:14.430 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 19 MB/s | 46 MB 00:02 2024-09-17T15:58:14.432 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:14.432 INFO:teuthology.orchestra.run.smithi006.stdout:Total 19 MB/s | 46 MB 00:02 2024-09-17T15:58:14.452 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:58:14.460 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:14.483 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:14.483 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:14.500 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:58:14.528 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:14.704 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): libcephsqlite-19.3.0-4975.g1889a165.el9. 757 kB/s | 162 kB 00:00 2024-09-17T15:58:14.869 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:14.869 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:15.137 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 470 kB/s | 304 kB 00:00 2024-09-17T15:58:15.163 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 139 kB/s | 64 kB 00:00 2024-09-17T15:58:15.393 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:15.481 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:58:15.530 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:58:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:15.913 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-19.3.0-4975.g1889a165.el9.x86_ 32 MB/s | 46 MB 00:01 2024-09-17T15:58:15.916 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:15.916 INFO:teuthology.orchestra.run.smithi181.stdout:Total 30 MB/s | 46 MB 00:01 2024-09-17T15:58:15.944 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:15.964 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:15.965 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:16.346 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:16.346 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:16.869 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:58:16.947 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-17T15:58:17.014 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-17T15:58:17.048 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:17.279 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:17.355 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:19.027 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:19.595 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:19.595 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:58:19.595 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:58:19.595 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:20.016 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:20.171 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:20.249 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:20.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-09-17T15:58:20.895 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:21.037 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:21.039 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:21.039 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:58:21.039 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:21.039 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:58:21.040 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:58:21.041 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:21.042 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:21.043 INFO:teuthology.orchestra.run.smithi050.stdout:Install 37 Packages 2024-09-17T15:58:21.043 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:21.043 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 M 2024-09-17T15:58:21.043 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 109 M 2024-09-17T15:58:21.043 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:21.376 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:21.376 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:58:21.377 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:58:21.377 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:58:21.410 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 3/4 2024-09-17T15:58:21.497 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:21.735 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:58:21.736 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:58:21.736 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:21.736 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:21.969 INFO:teuthology.orchestra.run.smithi050.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 2.7 MB/s | 1.5 MB 00:00 2024-09-17T15:58:22.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-09-17T15:58:22.094 INFO:teuthology.orchestra.run.smithi050.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 3.5 MB/s | 2.4 MB 00:00 2024-09-17T15:58:22.187 INFO:teuthology.orchestra.run.smithi050.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 3.1 MB/s | 291 kB 00:00 2024-09-17T15:58:22.421 INFO:teuthology.orchestra.run.smithi050.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 5.6 MB/s | 4.8 MB 00:00 2024-09-17T15:58:22.446 INFO:teuthology.orchestra.run.smithi050.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 979 kB/s | 253 kB 00:00 2024-09-17T15:58:22.496 INFO:teuthology.orchestra.run.smithi050.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T15:58:22.546 INFO:teuthology.orchestra.run.smithi050.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-17T15:58:22.605 INFO:teuthology.orchestra.run.smithi050.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-17T15:58:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:58:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:58:22.617 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:58:22.664 INFO:teuthology.orchestra.run.smithi050.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-17T15:58:22.676 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:39 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:22.731 INFO:teuthology.orchestra.run.smithi050.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2024-09-17T15:58:22.789 INFO:teuthology.orchestra.run.smithi050.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-17T15:58:22.831 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:22.831 INFO:teuthology.orchestra.run.smithi050.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 833 kB/s | 35 kB 00:00 2024-09-17T15:58:22.833 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:58:22.834 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:58:22.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:58:22.836 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout:Install 37 Packages 2024-09-17T15:58:22.837 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:22.838 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 31 M 2024-09-17T15:58:22.838 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 109 M 2024-09-17T15:58:22.838 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:22.856 INFO:teuthology.orchestra.run.smithi050.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-17T15:58:22.873 INFO:teuthology.orchestra.run.smithi050.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-17T15:58:22.898 INFO:teuthology.orchestra.run.smithi050.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-17T15:58:22.924 INFO:teuthology.orchestra.run.smithi050.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:22.961 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:58:22.962 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:58:22.962 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:22.962 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:22.974 INFO:teuthology.orchestra.run.smithi050.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-17T15:58:23.000 INFO:teuthology.orchestra.run.smithi050.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-17T15:58:23.025 INFO:teuthology.orchestra.run.smithi050.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-17T15:58:23.050 INFO:teuthology.orchestra.run.smithi050.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-17T15:58:23.075 INFO:teuthology.orchestra.run.smithi050.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 603 kB/s | 11 kB 00:00 2024-09-17T15:58:23.101 INFO:teuthology.orchestra.run.smithi050.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 766 kB/s | 18 kB 00:00 2024-09-17T15:58:23.118 INFO:teuthology.orchestra.run.smithi050.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-17T15:58:23.143 INFO:teuthology.orchestra.run.smithi050.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 784 kB/s | 19 kB 00:00 2024-09-17T15:58:23.168 INFO:teuthology.orchestra.run.smithi050.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 783 kB/s | 20 kB 00:00 2024-09-17T15:58:23.194 INFO:teuthology.orchestra.run.smithi050.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.8 MB/s | 46 kB 00:00 2024-09-17T15:58:23.222 INFO:teuthology.orchestra.run.smithi050.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-09-17T15:58:23.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2024-09-17T15:58:23.248 INFO:teuthology.orchestra.run.smithi050.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.4 MB/s | 79 kB 00:00 2024-09-17T15:58:23.273 INFO:teuthology.orchestra.run.smithi050.stdout:(30/37): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-09-17T15:58:23.298 INFO:teuthology.orchestra.run.smithi050.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-17T15:58:23.323 INFO:teuthology.orchestra.run.smithi050.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T15:58:23.349 INFO:teuthology.orchestra.run.smithi050.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T15:58:23.374 INFO:teuthology.orchestra.run.smithi050.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-17T15:58:23.399 INFO:teuthology.orchestra.run.smithi050.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-17T15:58:23.573 INFO:teuthology.orchestra.run.smithi169.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-09-17T15:58:23.574 INFO:teuthology.orchestra.run.smithi050.stdout:(36/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86 11 MB/s | 17 MB 00:01 2024-09-17T15:58:23.608 INFO:teuthology.orchestra.run.smithi050.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-09-17T15:58:23.611 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:23.611 INFO:teuthology.orchestra.run.smithi050.stdout:Total 12 MB/s | 31 MB 00:02 2024-09-17T15:58:23.756 INFO:teuthology.orchestra.run.smithi169.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-09-17T15:58:23.764 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:23.809 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:23.832 INFO:teuthology.orchestra.run.smithi169.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 5.7 MB/s | 2.4 MB 00:00 2024-09-17T15:58:23.848 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:23.876 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 4/4 2024-09-17T15:58:23.876 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 1/4 2024-09-17T15:58:23.876 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2/4 2024-09-17T15:58:23.876 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-17T15:58:23.882 INFO:teuthology.orchestra.run.smithi169.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 5.7 MB/s | 291 kB 00:00 2024-09-17T15:58:23.989 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:23.991 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:23.991 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:58:23.991 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:58:23.992 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:58:23.993 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:58:23.994 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout:Install 37 Packages 2024-09-17T15:58:23.995 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:23.996 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 31 M 2024-09-17T15:58:23.996 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 109 M 2024-09-17T15:58:23.996 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:24.066 INFO:teuthology.orchestra.run.smithi169.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 9.7 MB/s | 4.8 MB 00:00 2024-09-17T15:58:24.091 INFO:teuthology.orchestra.run.smithi169.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-17T15:58:24.150 INFO:teuthology.orchestra.run.smithi169.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-17T15:58:24.191 INFO:teuthology.orchestra.run.smithi169.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T15:58:24.200 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T15:58:24.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:24.201 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:24.217 INFO:teuthology.orchestra.run.smithi169.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 849 kB/s | 106 kB 00:00 2024-09-17T15:58:24.242 INFO:teuthology.orchestra.run.smithi169.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-17T15:58:24.292 INFO:teuthology.orchestra.run.smithi169.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-17T15:58:24.354 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:24.355 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:24.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-17T15:58:24.635 INFO:teuthology.orchestra.run.smithi169.stdout:(12/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86 20 MB/s | 17 MB 00:00 2024-09-17T15:58:24.701 INFO:teuthology.orchestra.run.smithi006.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-17T15:58:24.869 INFO:teuthology.orchestra.run.smithi006.stdout:(2/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-17T15:58:24.944 INFO:teuthology.orchestra.run.smithi006.stdout:(3/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.3 MB/s | 2.4 MB 00:00 2024-09-17T15:58:25.002 INFO:teuthology.orchestra.run.smithi006.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 4.9 MB/s | 291 kB 00:00 2024-09-17T15:58:25.068 INFO:teuthology.orchestra.run.smithi169.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 228 kB/s | 188 kB 00:00 2024-09-17T15:58:25.095 INFO:teuthology.orchestra.run.smithi006.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-17T15:58:25.097 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:40 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:25.168 INFO:teuthology.orchestra.run.smithi169.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 197 kB/s | 172 kB 00:00 2024-09-17T15:58:25.210 INFO:teuthology.orchestra.run.smithi169.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 61 kB/s | 35 kB 00:00 2024-09-17T15:58:25.239 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:25.241 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:25.241 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:58:25.241 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-4975.g1889a165.el9 ceph 6.6 k 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-4975.g1889a165.el9 ceph 2.4 M 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-4975.g1889a165.el9 ceph 1.5 M 2024-09-17T15:58:25.242 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 291 k 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-4975.g1889a165.el9 ceph 4.8 M 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 17 M 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T15:58:25.243 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T15:58:25.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T15:58:25.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T15:58:25.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T15:58:25.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T15:58:25.244 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T15:58:25.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T15:58:25.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-17T15:58:25.245 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-17T15:58:25.245 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T15:58:25.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T15:58:25.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T15:58:25.246 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T15:58:25.247 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T15:58:25.247 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T15:58:25.247 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T15:58:25.247 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T15:58:25.248 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T15:58:25.249 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T15:58:25.249 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T15:58:25.249 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T15:58:25.249 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:25.250 INFO:teuthology.orchestra.run.smithi181.stdout:Install 37 Packages 2024-09-17T15:58:25.251 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:25.252 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 M 2024-09-17T15:58:25.252 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 109 M 2024-09-17T15:58:25.252 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:25.495 INFO:teuthology.orchestra.run.smithi006.stdout:(6/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86_ 27 MB/s | 17 MB 00:00 2024-09-17T15:58:25.637 INFO:teuthology.orchestra.run.smithi006.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 399 kB/s | 253 kB 00:00 2024-09-17T15:58:25.694 INFO:teuthology.orchestra.run.smithi169.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 123 kB/s | 77 kB 00:00 2024-09-17T15:58:25.709 INFO:teuthology.orchestra.run.smithi181.stdout:(1/37): ceph-19.3.0-4975.g1889a165.el9.x86_64.r 50 kB/s | 6.6 kB 00:00 2024-09-17T15:58:25.754 INFO:teuthology.orchestra.run.smithi006.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-17T15:58:25.761 INFO:teuthology.orchestra.run.smithi169.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 86 kB/s | 51 kB 00:00 2024-09-17T15:58:25.802 INFO:teuthology.orchestra.run.smithi169.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-17T15:58:25.828 INFO:teuthology.orchestra.run.smithi169.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-17T15:58:25.853 INFO:teuthology.orchestra.run.smithi169.stdout:(20/37): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-09-17T15:58:25.870 INFO:teuthology.orchestra.run.smithi006.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.0 MB/s | 125 kB 00:00 2024-09-17T15:58:25.887 INFO:teuthology.orchestra.run.smithi169.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-17T15:58:25.899 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:25.912 INFO:teuthology.orchestra.run.smithi169.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-17T15:58:25.926 INFO:teuthology.orchestra.run.smithi181.stdout:(2/37): ceph-mds-19.3.0-4975.g1889a165.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-17T15:58:25.937 INFO:teuthology.orchestra.run.smithi169.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-09-17T15:58:25.954 INFO:teuthology.orchestra.run.smithi006.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 232 kB/s | 106 kB 00:00 2024-09-17T15:58:25.960 INFO:teuthology.orchestra.run.smithi181.stdout:(3/37): ceph-mgr-19.3.0-4975.g1889a165.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-09-17T15:58:25.961 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:58:25.971 INFO:teuthology.orchestra.run.smithi169.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 891 kB/s | 23 kB 00:00 2024-09-17T15:58:25.994 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:58:25.996 INFO:teuthology.orchestra.run.smithi169.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 782 kB/s | 19 kB 00:00 2024-09-17T15:58:26.004 INFO:teuthology.orchestra.run.smithi006.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-09-17T15:58:26.022 INFO:teuthology.orchestra.run.smithi169.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 782 kB/s | 20 kB 00:00 2024-09-17T15:58:26.030 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:58:26.047 INFO:teuthology.orchestra.run.smithi169.stdout:(27/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-09-17T15:58:26.053 INFO:teuthology.orchestra.run.smithi181.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4975.g1889 3.1 MB/s | 291 kB 00:00 2024-09-17T15:58:26.072 INFO:teuthology.orchestra.run.smithi169.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-17T15:58:26.097 INFO:teuthology.orchestra.run.smithi169.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.6 MB/s | 79 kB 00:00 2024-09-17T15:58:26.105 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:58:26.119 INFO:teuthology.orchestra.run.smithi181.stdout:(5/37): ceph-mon-19.3.0-4975.g1889a165.el9.x86_ 12 MB/s | 4.8 MB 00:00 2024-09-17T15:58:26.123 INFO:teuthology.orchestra.run.smithi169.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-17T15:58:26.129 INFO:teuthology.orchestra.run.smithi006.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-17T15:58:26.148 INFO:teuthology.orchestra.run.smithi169.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-17T15:58:26.173 INFO:teuthology.orchestra.run.smithi169.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-17T15:58:26.200 INFO:teuthology.orchestra.run.smithi169.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-09-17T15:58:26.224 INFO:teuthology.orchestra.run.smithi169.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-17T15:58:26.236 INFO:teuthology.orchestra.run.smithi181.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-17T15:58:26.249 INFO:teuthology.orchestra.run.smithi169.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-17T15:58:26.263 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:58:26.278 INFO:teuthology.orchestra.run.smithi181.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-17T15:58:26.283 INFO:teuthology.orchestra.run.smithi169.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-17T15:58:26.296 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:58:26.328 INFO:teuthology.orchestra.run.smithi181.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-09-17T15:58:26.333 INFO:teuthology.orchestra.run.smithi169.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-09-17T15:58:26.339 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:26.339 INFO:teuthology.orchestra.run.smithi169.stdout:Total 8.8 MB/s | 31 MB 00:03 2024-09-17T15:58:26.353 INFO:teuthology.orchestra.run.smithi181.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-17T15:58:26.379 INFO:teuthology.orchestra.run.smithi181.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-17T15:58:26.426 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:58:26.426 INFO:teuthology.orchestra.run.smithi181.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-17T15:58:26.469 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:58:26.493 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:26.524 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:58:26.534 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:26.534 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:26.559 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:58:26.588 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:58:26.622 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:58:26.629 INFO:teuthology.orchestra.run.smithi181.stdout:(12/37): ceph-osd-19.3.0-4975.g1889a165.el9.x86 24 MB/s | 17 MB 00:00 2024-09-17T15:58:26.656 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:58:26.680 INFO:teuthology.orchestra.run.smithi006.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 63 kB/s | 35 kB 00:00 2024-09-17T15:58:26.705 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:58:26.721 INFO:teuthology.orchestra.run.smithi181.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 551 kB/s | 188 kB 00:00 2024-09-17T15:58:26.730 INFO:teuthology.orchestra.run.smithi006.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 237 kB/s | 172 kB 00:00 2024-09-17T15:58:26.746 INFO:teuthology.orchestra.run.smithi181.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 503 kB/s | 172 kB 00:00 2024-09-17T15:58:26.748 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:58:26.772 INFO:teuthology.orchestra.run.smithi181.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-09-17T15:58:26.781 INFO:teuthology.orchestra.run.smithi006.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 228 kB/s | 188 kB 00:00 2024-09-17T15:58:26.782 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:58:26.814 INFO:teuthology.orchestra.run.smithi181.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 837 kB/s | 77 kB 00:00 2024-09-17T15:58:26.832 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:58:26.847 INFO:teuthology.orchestra.run.smithi181.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 556 kB/s | 42 kB 00:00 2024-09-17T15:58:26.872 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:58:26.873 INFO:teuthology.orchestra.run.smithi181.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 508 kB/s | 51 kB 00:00 2024-09-17T15:58:26.898 INFO:teuthology.orchestra.run.smithi181.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 266 kB/s | 22 kB 00:00 2024-09-17T15:58:26.948 INFO:teuthology.orchestra.run.smithi181.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-09-17T15:58:26.949 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:58:26.973 INFO:teuthology.orchestra.run.smithi181.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-17T15:58:26.999 INFO:teuthology.orchestra.run.smithi181.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-17T15:58:27.015 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:27.015 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:27.024 INFO:teuthology.orchestra.run.smithi181.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 711 kB/s | 18 kB 00:00 2024-09-17T15:58:27.049 INFO:teuthology.orchestra.run.smithi181.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-17T15:58:27.075 INFO:teuthology.orchestra.run.smithi181.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-17T15:58:27.100 INFO:teuthology.orchestra.run.smithi181.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T15:58:27.102 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:58:27.125 INFO:teuthology.orchestra.run.smithi181.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-17T15:58:27.150 INFO:teuthology.orchestra.run.smithi181.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-17T15:58:27.176 INFO:teuthology.orchestra.run.smithi181.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-17T15:58:27.191 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:58:27.201 INFO:teuthology.orchestra.run.smithi181.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-17T15:58:27.226 INFO:teuthology.orchestra.run.smithi181.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-17T15:58:27.245 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:58:27.251 INFO:teuthology.orchestra.run.smithi181.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-17T15:58:27.277 INFO:teuthology.orchestra.run.smithi181.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-17T15:58:27.283 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:58:27.311 INFO:teuthology.orchestra.run.smithi181.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-17T15:58:27.336 INFO:teuthology.orchestra.run.smithi181.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 807 kB/s | 20 kB 00:00 2024-09-17T15:58:27.339 INFO:teuthology.orchestra.run.smithi006.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 84 kB/s | 51 kB 00:00 2024-09-17T15:58:27.342 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:58:27.361 INFO:teuthology.orchestra.run.smithi181.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-17T15:58:27.364 INFO:teuthology.orchestra.run.smithi006.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 71 kB/s | 42 kB 00:00 2024-09-17T15:58:27.389 INFO:teuthology.orchestra.run.smithi006.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 109 kB/s | 77 kB 00:00 2024-09-17T15:58:27.394 INFO:teuthology.orchestra.run.smithi181.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-17T15:58:27.399 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:27.399 INFO:teuthology.orchestra.run.smithi181.stdout:Total 14 MB/s | 31 MB 00:02 2024-09-17T15:58:27.405 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:58:27.440 INFO:teuthology.orchestra.run.smithi006.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-17T15:58:27.473 INFO:teuthology.orchestra.run.smithi006.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-09-17T15:58:27.499 INFO:teuthology.orchestra.run.smithi006.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-17T15:58:27.524 INFO:teuthology.orchestra.run.smithi006.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-17T15:58:27.549 INFO:teuthology.orchestra.run.smithi006.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 705 kB/s | 18 kB 00:00 2024-09-17T15:58:27.553 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:27.574 INFO:teuthology.orchestra.run.smithi006.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 930 kB/s | 23 kB 00:00 2024-09-17T15:58:27.594 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:27.594 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:27.600 INFO:teuthology.orchestra.run.smithi006.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-17T15:58:27.611 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:58:27.625 INFO:teuthology.orchestra.run.smithi006.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-17T15:58:27.650 INFO:teuthology.orchestra.run.smithi006.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-17T15:58:27.670 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:58:27.675 INFO:teuthology.orchestra.run.smithi006.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-17T15:58:27.701 INFO:teuthology.orchestra.run.smithi006.stdout:(29/37): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-09-17T15:58:27.734 INFO:teuthology.orchestra.run.smithi006.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 940 kB/s | 79 kB 00:00 2024-09-17T15:58:27.752 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:58:27.760 INFO:teuthology.orchestra.run.smithi006.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-09-17T15:58:27.793 INFO:teuthology.orchestra.run.smithi006.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.6 MB/s | 90 kB 00:00 2024-09-17T15:58:27.819 INFO:teuthology.orchestra.run.smithi006.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 611 kB/s | 36 kB 00:00 2024-09-17T15:58:27.841 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:58:27.844 INFO:teuthology.orchestra.run.smithi006.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-17T15:58:27.869 INFO:teuthology.orchestra.run.smithi006.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-17T15:58:27.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:58:27.894 INFO:teuthology.orchestra.run.smithi006.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-17T15:58:27.920 INFO:teuthology.orchestra.run.smithi006.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-17T15:58:27.924 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:27.925 INFO:teuthology.orchestra.run.smithi006.stdout:Total 7.8 MB/s | 31 MB 00:03 2024-09-17T15:58:28.024 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:58:28.084 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:28.085 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:28.085 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:28.129 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:28.129 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:28.199 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:58:28.229 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:28.257 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:28.399 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:28.461 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:58:28.514 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:58:28.547 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:58:28.618 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:58:28.631 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:28.631 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:28.663 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:58:28.703 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:58:28.821 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:58:28.899 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:58:28.949 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:58:28.983 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:58:29.022 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:58:29.057 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:58:29.091 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:58:29.131 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:58:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:58:29.217 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:58:29.265 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:58:29.297 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:58:29.370 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:58:29.446 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:58:29.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:58:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:58:29.570 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:58:29.589 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:58:29.608 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:58:29.650 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:58:29.679 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:58:29.697 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:58:29.732 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:58:29.746 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:58:29.773 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:58:29.802 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:58:29.805 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:29.834 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:29.835 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:29.835 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:58:29.835 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:29.835 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:29.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:29.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:58:29.943 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:58:29.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:58:29.989 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:58:30.002 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:30.019 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:58:30.043 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:58:30.057 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:58:30.081 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-17T15:58:30.085 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:58:30.121 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:58:30.126 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:58:30.128 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-17T15:58:30.174 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-17T15:58:30.184 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:58:30.212 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:58:30.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:58:30.253 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:58:30.261 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:58:30.261 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-17T15:58:30.300 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-17T15:58:30.301 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:58:30.314 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:30.335 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:58:30.340 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:30.340 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:30.340 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:58:30.340 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:30.341 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:30.341 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:30.342 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-17T15:58:30.389 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:58:30.416 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:58:30.456 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-17T15:58:30.541 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-17T15:58:30.549 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:58:30.552 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:58:30.573 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:30.586 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-17T15:58:30.588 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:30.598 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:30.598 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:30.598 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:58:30.598 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:30.599 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:30.599 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:30.614 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:30.615 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:30.615 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:58:30.615 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:30.615 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:30.615 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:30.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-17T15:58:30.633 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:58:30.650 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-17T15:58:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:30.685 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-17T15:58:30.694 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:58:30.718 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-17T15:58:30.734 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:58:30.760 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-17T15:58:30.783 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:58:30.811 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-17T15:58:30.838 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:58:30.845 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-17T15:58:30.895 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-17T15:58:30.935 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-17T15:58:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-17T15:58:31.023 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:58:31.071 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:58:31.149 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:58:31.164 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-17T15:58:31.237 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:58:31.245 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-17T15:58:31.270 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:58:31.322 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-17T15:58:31.367 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-17T15:58:31.407 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:58:31.428 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-17T15:58:31.491 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-17T15:58:31.576 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:58:31.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:31.633 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:31.688 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-17T15:58:31.738 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-17T15:58:31.820 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-17T15:58:31.918 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-17T15:58:31.954 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-17T15:58:32.072 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:32.099 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-17T15:58:32.099 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:32.099 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:32.099 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:58:32.099 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:32.100 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:32.100 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:32.275 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 32/37 2024-09-17T15:58:32.314 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:32.341 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 33/37 2024-09-17T15:58:32.341 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:32.341 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T15:58:32.341 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:32.341 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T15:58:32.342 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:58:32.442 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:58:32.443 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:58:32.444 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:58:32.445 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:58:32.556 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:32.581 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:32.581 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:32.581 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:58:32.581 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:32.581 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:32.582 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:32.807 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:32.830 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:32.830 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:32.830 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:58:32.831 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:32.831 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:32.831 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:32.894 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:33.149 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:33.176 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:33.177 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:33.177 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:58:33.177 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:33.177 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:33.177 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:33.568 INFO:teuthology.orchestra.run.smithi050.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:58:33.569 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:58:33.570 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:33.571 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:33.892 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:33.915 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:33.915 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:33.915 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:58:33.916 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:33.916 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:33.916 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:33.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 34/37 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T15:58:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:33.980 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:34.407 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 35/37 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T15:58:34.432 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:34.538 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:34.631 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:34.674 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:34.675 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:34.676 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:58:34.677 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:34.699 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 36/37 2024-09-17T15:58:34.699 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:34.700 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T15:58:34.700 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:34.700 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T15:58:34.700 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:34.713 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:34.713 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:58:34.713 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:58:34.714 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:58:34.715 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:58:34.716 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:58:34.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-09-17T15:58:34.774 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:35.357 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:35.493 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:35.494 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:35.494 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:35.495 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 784 k 2024-09-17T15:58:35.496 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 791 k 2024-09-17T15:58:35.496 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:35.666 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:58:35.666 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:35.666 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:58:35.667 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:58:35.668 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:35.669 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:35.685 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:35.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:58:35.685 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:58:35.686 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:58:35.687 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:58:35.689 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:58:35.690 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:58:35.690 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:58:35.771 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-17T15:58:35.772 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:35.772 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-17T15:58:35.772 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:35.778 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:35.778 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:35.784 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:35.785 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:35.872 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:36.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-09-17T15:58:36.261 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:36.353 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:36.599 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 37/37 2024-09-17T15:58:36.599 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.3.0-4975.g1889a165.el9.x86_64 1/37 2024-09-17T15:58:36.599 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 3/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 4/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 5/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9. 6/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-17T15:58:36.600 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-17T15:58:36.601 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-17T15:58:36.602 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-17T15:58:36.603 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-17T15:58:36.731 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:36.732 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:36.822 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:36.838 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:58:36.839 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:58:36.840 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:36.841 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:36.865 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:36.865 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:58:36.865 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:36.939 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:36.973 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:37.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-17T15:58:37.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:58:37.530 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:54 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:37.627 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-17T15:58:37.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:37.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:37.627 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T15:58:37.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T15:58:37.629 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:37.630 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:37.661 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:37.662 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:37.662 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:58:37.662 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:37.662 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:37.662 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 784 k 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 791 k 2024-09-17T15:58:37.663 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:37.778 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:52 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:37.808 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:37.867 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:37.909 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:37.910 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:58:37.910 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:37.923 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-17T15:58:37.923 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:37.924 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-17T15:58:37.924 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:37.929 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:37.929 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:37.935 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:37.935 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:37.946 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:37.947 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:37.948 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:58:37.948 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:37.948 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 145 k 2024-09-17T15:58:37.948 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 432 k 2024-09-17T15:58:37.948 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:37.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-17T15:58:38.011 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:38.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2024-09-17T15:58:38.152 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 710 kB/s | 145 kB 00:00 2024-09-17T15:58:38.153 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:38.153 INFO:teuthology.orchestra.run.smithi050.stdout:Total 704 kB/s | 145 kB 00:00 2024-09-17T15:58:38.153 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:38.161 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:38.162 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:38.222 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:38.222 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:38.428 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:38.455 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:38.455 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:58:38.455 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:38.517 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:38.561 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:38.611 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:38.646 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:55 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:38.691 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 784 k 2024-09-17T15:58:38.692 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 791 k 2024-09-17T15:58:38.693 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:38.738 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:38.783 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:38.784 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:58:38.785 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:38.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2024-09-17T15:58:38.953 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-17T15:58:38.953 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:38.953 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-17T15:58:38.954 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:38.959 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:38.959 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:38.964 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:38.965 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:38.977 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:39.041 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:58:39.249 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:39.249 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:39.250 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:39.250 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:39.250 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:39.250 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:39.406 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:39.459 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:39.486 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:58:39.498 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:39.598 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 784 k 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 784 k 2024-09-17T15:58:39.599 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 791 k 2024-09-17T15:58:39.600 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:39.614 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:39.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-09-17T15:58:39.874 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-19.3.0-4975.g1889a165.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-17T15:58:39.874 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:39.874 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-17T15:58:39.874 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:39.880 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:39.880 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:39.887 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:39.887 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:39.962 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:39.975 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:40.085 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:57 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:40.222 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:40.223 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:40.223 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-17T15:58:40.223 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:40.223 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 145 k 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 432 k 2024-09-17T15:58:40.224 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:40.230 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:40.230 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:40.230 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:40.231 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:40.231 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:40.231 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:40.239 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:40.332 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:40.375 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:40.376 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:58:40.376 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:40.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:58:40.471 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:40.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:58:40.521 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 488 kB/s | 145 kB 00:00 2024-09-17T15:58:40.521 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:40.522 INFO:teuthology.orchestra.run.smithi169.stdout:Total 486 kB/s | 145 kB 00:00 2024-09-17T15:58:40.522 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:40.530 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:40.530 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:40.582 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:40.586 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:40.587 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:40.729 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:40.789 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:40.814 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:40.814 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:40.814 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:58:40.814 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:40.891 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:41.049 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:01 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:41.071 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:41.151 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:58:41.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:41.152 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:41.152 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:41.152 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:41.152 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:41.190 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:58:41.192 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-17T15:58:41.193 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:41.194 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 19 M 2024-09-17T15:58:41.194 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 159 M 2024-09-17T15:58:41.194 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:41.205 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:58:41.206 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:41.207 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-09-17T15:58:41.207 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 432 k 2024-09-17T15:58:41.207 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:41.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2024-09-17T15:58:41.418 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 686 kB/s | 145 kB 00:00 2024-09-17T15:58:41.419 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:41.419 INFO:teuthology.orchestra.run.smithi181.stdout:Total 681 kB/s | 145 kB 00:00 2024-09-17T15:58:41.419 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:41.427 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:41.427 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:41.483 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:41.483 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:41.626 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:58:41.686 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:41.711 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:41.711 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:41.711 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:58:41.711 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:41.781 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 197 kB/s | 34 kB 00:00 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:41.802 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:41.806 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 82 kB/s | 16 kB 00:00 2024-09-17T15:58:41.963 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:42.033 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-09-17T15:58:42.101 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:42.101 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4975.g1889a165.el9 ceph 145 k 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:58:42.102 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:42.103 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 145 k 2024-09-17T15:58:42.103 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 432 k 2024-09-17T15:58:42.103 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:42.315 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-19.3.0-4975.g1889a1 683 kB/s | 145 kB 00:00 2024-09-17T15:58:42.316 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:42.316 INFO:teuthology.orchestra.run.smithi006.stdout:Total 678 kB/s | 145 kB 00:00 2024-09-17T15:58:42.316 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:42.324 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:42.324 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:42.384 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:42.385 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:42.535 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:42.574 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 351 kB/s | 269 kB 00:00 2024-09-17T15:58:42.616 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:42.628 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:42.638 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:42.641 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:42.641 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:58:42.641 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T15:58:42.642 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:42.716 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.1 MB/s | 1.0 MB 00:00 2024-09-17T15:58:42.721 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:42.765 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:42.766 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:58:42.766 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:42.774 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.3 MB/s | 548 kB 00:00 2024-09-17T15:58:42.799 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-09-17T15:58:42.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:58:42.866 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-17T15:58:42.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-17T15:58:42.891 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-17T15:58:42.917 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-09-17T15:58:42.942 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-09-17T15:58:43.117 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 9.4 MB/s | 14 MB 00:01 2024-09-17T15:58:43.142 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 837 kB/s | 188 kB 00:00 2024-09-17T15:58:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.9 MB/s | 19 MB 00:01 2024-09-17T15:58:43.201 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:43.223 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:43.223 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:43.429 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:00 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:43.471 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:58 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:43.525 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:43.525 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:43.560 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:43.561 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:43.561 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:58:43.562 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-09-17T15:58:43.563 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:43.564 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:43.564 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 19 M 2024-09-17T15:58:43.564 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 159 M 2024-09-17T15:58:43.564 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:43.606 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:43.609 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:58:43.609 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:43.687 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4975.g1889a165. 1/1 2024-09-17T15:58:43.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:43.687 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:43.687 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:58:43.687 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:43.688 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:43.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:58:43.809 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 230 kB/s | 34 kB 00:00 2024-09-17T15:58:43.834 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 95 kB/s | 16 kB 00:00 2024-09-17T15:58:43.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2024-09-17T15:58:43.968 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 269 kB 00:00 2024-09-17T15:58:44.026 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-17T15:58:44.085 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T15:58:44.110 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-09-17T15:58:44.132 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:44.177 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-17T15:58:44.202 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-17T15:58:44.202 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:58:44.227 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T15:58:44.253 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-17T15:58:44.283 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:59 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:58:44.361 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 2.1 MB/s | 863 kB 00:00 2024-09-17T15:58:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:58:44.418 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:44.419 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:44.419 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:58:44.420 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-17T15:58:44.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:44.422 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 19 M 2024-09-17T15:58:44.422 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 159 M 2024-09-17T15:58:44.422 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:58:44.532 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:58:44.549 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:01 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:44.636 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4975.g1889a1 15 MB/s | 14 MB 00:00 2024-09-17T15:58:44.640 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:44.640 INFO:teuthology.orchestra.run.smithi169.stdout:Total 18 MB/s | 19 MB 00:01 2024-09-17T15:58:44.642 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:58:44.663 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:58:44.688 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:44.688 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:44.689 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:58:44.689 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:44.707 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:44.708 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:44.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2024-09-17T15:58:44.764 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:58:44.812 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:58:44.839 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:44.871 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:58:44.969 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:44.969 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:44.989 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 231 kB/s | 34 kB 00:00 2024-09-17T15:58:45.014 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 95 kB/s | 16 kB 00:00 2024-09-17T15:58:45.359 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:02 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:45.501 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 14 M 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T15:58:45.502 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 34 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 16 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:58:45.503 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-09-17T15:58:45.504 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:45.504 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 19 M 2024-09-17T15:58:45.504 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 159 M 2024-09-17T15:58:45.504 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:45.555 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:45.607 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4975.g1889a16 19 MB/s | 14 MB 00:00 2024-09-17T15:58:45.637 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:58:45.640 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 430 kB/s | 269 kB 00:00 2024-09-17T15:58:45.680 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:58:45.699 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T15:58:45.716 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-17T15:58:45.726 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:58:45.758 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:58:45.774 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-17T15:58:45.799 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-17T15:58:45.825 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T15:58:45.859 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-17T15:58:45.867 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:58:45.884 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-17T15:58:46.002 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:58:46.094 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:58:46.154 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:58:46.191 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:46.202 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4975.g18 228 kB/s | 34 kB 00:00 2024-09-17T15:58:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:58:46.227 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4975.g188 94 kB/s | 16 kB 00:00 2024-09-17T15:58:46.550 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 915 kB/s | 863 kB 00:00 2024-09-17T15:58:46.554 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:46.554 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.0 MB/s | 19 MB 00:02 2024-09-17T15:58:46.606 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:46.625 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:46.626 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:46.820 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): ceph-mgr-dashboard-19.3.0-4975.g1889a16 18 MB/s | 14 MB 00:00 2024-09-17T15:58:46.904 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:46.905 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:46.936 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 380 kB/s | 269 kB 00:00 2024-09-17T15:58:46.987 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:58:47.012 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 782 kB/s | 19 kB 00:00 2024-09-17T15:58:47.037 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-09-17T15:58:47.088 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-09-17T15:58:47.113 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-17T15:58:47.138 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-17T15:58:47.180 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-17T15:58:47.288 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 952 kB/s | 1.0 MB 00:01 2024-09-17T15:58:47.293 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:47.293 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 19 MB 00:01 2024-09-17T15:58:47.348 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:47.370 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:47.370 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:47.504 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:58:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:58:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:58:47.662 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:47.662 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:58:47.705 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:58:47.805 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:58:47.938 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:58:48.033 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:58:48.076 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:58:48.105 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:48.137 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:58:48.259 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:58:48.283 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:48.317 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:48.365 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-17T15:58:48.409 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-17T15:58:48.456 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-17T15:58:48.493 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-17T15:58:48.609 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-17T15:58:48.742 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-17T15:58:48.840 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-17T15:58:48.899 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-17T15:58:48.926 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:48.959 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 10/12 2024-09-17T15:58:49.429 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:58:49.503 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:50.081 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:50.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:58:50.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:58:50.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:58:50.081 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:58:50.082 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:50.409 INFO:teuthology.orchestra.run.smithi050.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:50.410 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:58:50.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:58:51.277 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:11 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:58:51.297 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:58:51.359 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:51.418 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:58:51.419 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-17T15:58:51.419 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:58:51.419 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-17T15:58:51.419 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:58:51.419 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:58:51.420 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:======================================================================================= 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 212 M 2024-09-17T15:58:51.421 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:58:51.470 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:58:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:58:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:58:51.471 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:58:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:58:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:51.788 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:58:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:51.789 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:58:51.936 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-49 16 MB/s | 7.4 MB 00:00 2024-09-17T15:58:52.050 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 288 kB/s | 33 kB 00:00 2024-09-17T15:58:52.065 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:58:52.217 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 254 kB/s | 191 kB 00:00 2024-09-17T15:58:52.334 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 156 kB/s | 18 kB 00:00 2024-09-17T15:58:52.352 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 11/12 2024-09-17T15:58:52.404 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:52.435 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 385 kB/s | 38 kB 00:00 2024-09-17T15:58:52.468 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 799 kB/s | 801 kB 00:01 2024-09-17T15:58:52.657 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:52.702 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-17T15:58:52.800 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:58:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2024-09-17T15:58:52.802 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-09-17T15:58:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:58:52.803 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 38 M 2024-09-17T15:58:52.803 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 212 M 2024-09-17T15:58:52.803 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:58:53.085 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-09-17T15:58:53.227 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:58:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:58:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:58:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:58:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:58:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:58:53.377 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-17T15:58:53.461 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 395 kB/s | 33 kB 00:00 2024-09-17T15:58:53.553 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:58:53.553 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:53.553 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:58:53.553 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:58:53.553 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:58:53.554 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:58:53.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:53.555 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:58:53.661 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-09-17T15:58:53.703 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-17T15:58:53.745 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 924 kB/s | 38 kB 00:00 2024-09-17T15:58:53.770 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): libquadmath-11.5.0-2.el9.x86_64.rpm 213 kB/s | 191 kB 00:00 2024-09-17T15:58:53.822 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:58:53.870 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 40 MB/s | 4.9 MB 00:00 2024-09-17T15:58:53.938 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.3 MB/s | 4.9 MB 00:01 2024-09-17T15:58:53.979 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 47 MB/s | 5.1 MB 00:00 2024-09-17T15:58:54.004 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-09-17T15:58:54.038 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): libgfortran-11.5.0-2.el9.x86_64.rpm 687 kB/s | 801 kB 00:01 2024-09-17T15:58:54.063 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.9 MB/s | 250 kB 00:00 2024-09-17T15:58:54.193 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 12/12 2024-09-17T15:58:54.193 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el 1/12 2024-09-17T15:58:54.193 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noa 2/12 2024-09-17T15:58:54.193 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9 3/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-17T15:58:54.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-17T15:58:54.330 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-09-17T15:58:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:54.331 INFO:teuthology.orchestra.run.smithi169.stdout:Total 25 MB/s | 38 MB 00:01 2024-09-17T15:58:54.411 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:09 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:58:54.420 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.0 MB/s | 5.1 MB 00:01 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T15:58:54.534 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:54.535 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:58:54.553 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:58:54.554 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-17T15:58:54.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:58:54.556 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-17T15:58:54.556 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-17T15:58:54.556 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:58:54.682 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:58:54.707 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:58:54.708 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:58:54.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-17T15:58:54.800 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 953 kB/s | 191 kB 00:00 2024-09-17T15:58:54.892 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-17T15:58:54.917 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-17T15:58:55.009 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-09-17T15:58:55.131 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:58:55.131 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:58:55.134 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): ceph-mgr-diskprediction-local-19.3.0-49 14 MB/s | 7.4 MB 00:00 2024-09-17T15:58:55.159 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 256 kB/s | 38 kB 00:00 2024-09-17T15:58:55.293 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-17T15:58:55.371 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:12 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:58:55.508 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:58:55.509 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-09-17T15:58:55.509 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 7.4 M 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T15:58:55.510 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================= 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:58:55.511 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 38 M 2024-09-17T15:58:55.512 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 212 M 2024-09-17T15:58:55.512 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:58:55.712 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-17T15:58:55.744 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.5 MB/s | 16 MB 00:02 2024-09-17T15:58:55.745 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:55.746 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.8 MB/s | 38 MB 00:04 2024-09-17T15:58:55.754 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 785 kB/s | 33 kB 00:00 2024-09-17T15:58:55.779 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.9 MB/s | 801 kB 00:00 2024-09-17T15:58:55.821 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 18 kB 00:00 2024-09-17T15:58:55.897 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 21 MB/s | 3.0 MB 00:00 2024-09-17T15:58:55.904 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:58:55.922 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 383 kB/s | 38 kB 00:00 2024-09-17T15:58:55.973 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-17T15:58:55.993 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:58:56.045 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:58:56.048 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 33 MB/s | 4.9 MB 00:00 2024-09-17T15:58:56.052 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-17T15:58:56.090 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-17T15:58:56.112 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:58:56.140 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:58:56.140 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:58:56.143 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:58:56.179 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:58:56.194 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-17T15:58:56.206 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): ceph-mgr-diskprediction-local-19.3.0-4 12 MB/s | 7.4 MB 00:00 2024-09-17T15:58:56.290 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-09-17T15:58:56.364 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:58:56.395 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:58:56.471 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:58:56.502 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.6 MB/s | 4.9 MB 00:01 2024-09-17T15:58:56.557 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-09-17T15:58:56.558 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:56.558 INFO:teuthology.orchestra.run.smithi006.stdout:Total 36 MB/s | 38 MB 00:01 2024-09-17T15:58:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:58:56.591 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:58:56.619 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-17T15:58:56.921 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:58:56.948 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:58:56.948 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:58:57.122 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:58:57.233 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:58:57.378 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:58:57.404 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:58:57.404 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:58:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:58:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:58:57.614 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:58:57.652 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:58:57.836 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:58:57.868 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:58:57.946 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:58:58.194 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:58:58.261 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:58:58.297 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:58:58.440 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:58:58.489 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:58:58.633 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:58:58.643 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:58:58.652 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-17T15:58:58.653 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:58:58.654 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.3 MB/s | 38 MB 00:04 2024-09-17T15:58:58.664 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:58:58.746 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:58:58.768 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:58:59.009 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:58:59.036 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:58:59.036 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:58:59.440 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:58:59.469 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:58:59.469 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:58:59.540 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:58:59.556 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:58:59.569 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:00.251 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:00.322 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-17T15:59:00.359 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:59:00.460 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:59:00.502 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-17T15:59:00.686 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-17T15:59:00.717 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-17T15:59:00.794 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-17T15:59:00.889 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:01.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:59:01.217 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:01.448 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-17T15:59:01.555 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-17T15:59:01.982 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:59:02.012 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:02.192 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:59:02.585 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:02.586 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:02.800 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:03.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:59:03.626 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:20 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:03.732 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:03.732 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:59:03.732 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:59:03.732 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:59:03.732 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:03.733 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:59:03.763 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:59:03.765 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-09-17T15:59:03.766 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:03.767 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.2 M 2024-09-17T15:59:03.767 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-09-17T15:59:03.767 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:03.829 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-17T15:59:03.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:04.188 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:59:04.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:04.188 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:04.188 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:04.189 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:04.287 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 223 kB/s | 19 kB 00:00 2024-09-17T15:59:04.312 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 230 kB/s | 26 kB 00:00 2024-09-17T15:59:04.370 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 290 kB/s | 49 kB 00:00 2024-09-17T15:59:04.404 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-17T15:59:04.429 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-17T15:59:04.454 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T15:59:04.497 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-17T15:59:04.522 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:59:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:59:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:59:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:59:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:59:04.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:59:04.547 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-17T15:59:04.581 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-17T15:59:04.606 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-17T15:59:04.623 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-09-17T15:59:04.665 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-17T15:59:04.668 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:04.668 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-17T15:59:04.736 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:04.752 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:04.752 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:04.839 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:04.841 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:59:05.066 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:05.067 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:05.153 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:25 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:05.292 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:05.293 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:05.293 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:59:05.293 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:05.293 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:05.293 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:59:05.294 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:05.295 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-09-17T15:59:05.296 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-09-17T15:59:05.296 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:05.341 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:05.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:59:05.520 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:59:05.574 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:59:05.630 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:59:05.667 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:59:05.708 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:59:05.818 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:59:05.850 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 254 kB/s | 49 kB 00:00 2024-09-17T15:59:05.925 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 95 kB/s | 26 kB 00:00 2024-09-17T15:59:05.928 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:59:05.951 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 65 kB/s | 19 kB 00:00 2024-09-17T15:59:05.993 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:59:06.091 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:23 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:06.231 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:06.232 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:06.232 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:59:06.232 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:59:06.233 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:06.234 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.2 M 2024-09-17T15:59:06.235 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-09-17T15:59:06.235 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:06.376 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 423 kB/s | 222 kB 00:00 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 12/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a 1/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-17T15:59:06.431 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-17T15:59:06.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-17T15:59:06.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-17T15:59:06.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-17T15:59:06.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-17T15:59:06.432 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-17T15:59:06.434 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 312 kB/s | 159 kB 00:00 2024-09-17T15:59:06.460 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 548 kB/s | 279 kB 00:00 2024-09-17T15:59:06.502 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-17T15:59:06.527 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 212 kB/s | 14 kB 00:00 2024-09-17T15:59:06.552 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.5 MB/s | 220 kB 00:00 2024-09-17T15:59:06.577 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T15:59:06.603 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-17T15:59:06.645 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-09-17T15:59:06.711 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:59:06.736 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 149 kB/s | 54 kB 00:00 2024-09-17T15:59:06.739 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:06.740 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-09-17T15:59:06.746 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 272 kB/s | 19 kB 00:00 2024-09-17T15:59:06.764 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 292 kB/s | 26 kB 00:00 2024-09-17T15:59:06.766 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:59:06.799 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:59:06.813 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:06.821 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-09-17T15:59:06.829 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-17T15:59:06.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:06.829 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:06.829 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:06.829 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:06.830 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:06.831 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:06.831 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:06.847 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 222 kB 00:00 2024-09-17T15:59:06.847 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:59:06.869 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:06.872 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 250 kB/s | 49 kB 00:00 2024-09-17T15:59:06.897 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-17T15:59:06.923 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-17T15:59:06.930 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:06.931 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:06.973 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-09-17T15:59:06.998 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-17T15:59:07.032 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-17T15:59:07.057 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-17T15:59:07.082 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-17T15:59:07.116 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.3 MB/s | 1.0 MB 00:00 2024-09-17T15:59:07.119 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:07.119 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.5 MB/s | 2.2 MB 00:00 2024-09-17T15:59:07.189 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:07.206 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:07.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-17T15:59:07.303 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:07.305 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:07.446 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:07.618 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:59:07.640 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:07.640 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:59:07.640 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:59:07.640 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:59:07.641 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:59:07.673 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:59:07.737 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:59:07.798 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:59:07.803 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:22 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:07.840 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:59:07.936 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:07.937 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:07.937 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:07.937 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 49 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T15:59:07.938 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-17T15:59:07.939 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:07.940 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-17T15:59:07.940 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-17T15:59:07.940 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:07.947 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:59:07.991 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:07.992 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:08.057 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:59:08.106 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:59:08.109 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:59:08.169 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:59:08.208 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): ceph-mgr-rook-19.3.0-4975.g1889a165.el9 306 kB/s | 49 kB 00:00 2024-09-17T15:59:08.224 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:59:08.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:59:08.268 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:59:08.311 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:59:08.334 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-09-17T15:59:08.359 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 82 kB/s | 26 kB 00:00 2024-09-17T15:59:08.418 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:59:08.536 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:59:08.588 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:59:08.701 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 433 kB/s | 159 kB 00:00 2024-09-17T15:59:08.759 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 403 kB/s | 222 kB 00:00 2024-09-17T15:59:08.801 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 772 kB/s | 32 kB 00:00 2024-09-17T15:59:08.826 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 427 kB/s | 54 kB 00:00 2024-09-17T15:59:08.837 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:25 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:08.852 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 566 kB/s | 279 kB 00:00 2024-09-17T15:59:08.877 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-09-17T15:59:08.881 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:59:08.930 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:59:08.944 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 17 MB/s | 1.0 MB 00:00 2024-09-17T15:59:08.963 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:59:08.969 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.5 MB/s | 220 kB 00:00 2024-09-17T15:59:08.971 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:59:08.972 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-09-17T15:59:08.973 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:08.974 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 7.0 M 2024-09-17T15:59:08.974 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-09-17T15:59:08.974 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:08.994 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 537 kB/s | 59 kB 00:00 2024-09-17T15:59:09.012 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:59:09.020 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-09-17T15:59:09.023 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:09.023 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.0 MB/s | 2.2 MB 00:01 2024-09-17T15:59:09.041 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:09.087 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:09.102 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:09.102 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:09.191 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:09.193 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:09.267 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-17T15:59:09.325 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T15:59:09.350 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:59:09.351 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T15:59:09.376 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 601 kB/s | 154 kB 00:00 2024-09-17T15:59:09.400 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:59:09.401 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-17T15:59:09.434 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:59:09.483 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:59:09.511 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:09.693 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:59:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:59:09.793 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:59:09.794 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:59:09.866 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-17T15:59:09.929 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-17T15:59:09.993 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-17T15:59:10.030 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-17T15:59:10.071 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:59:10.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:10.138 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:10.191 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-17T15:59:10.304 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-17T15:59:10.318 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:10.318 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:59:10.318 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:59:10.318 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:59:10.319 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:59:10.365 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-17T15:59:10.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:59:10.643 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-17T15:59:10.645 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:10.645 INFO:teuthology.orchestra.run.smithi169.stdout:Total 4.2 MB/s | 7.0 MB 00:01 2024-09-17T15:59:10.656 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:59:10.656 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:10.656 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:10.656 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:10.656 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:10.657 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:10.719 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:10.731 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:10.732 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:10.833 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:10.833 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:10.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:59:10.995 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:11.073 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-17T15:59:11.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-17T15:59:11.136 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:59:11.137 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-09-17T15:59:11.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:11.139 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-09-17T15:59:11.139 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-09-17T15:59:11.140 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:11.154 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-17T15:59:11.168 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:11.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-17T15:59:11.224 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:11.349 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:59:11.403 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:59:11.422 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-17T15:59:11.480 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-17T15:59:11.506 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 650 kB/s | 154 kB 00:00 2024-09-17T15:59:11.522 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:28 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:11.531 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-17T15:59:11.573 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-17T15:59:11.662 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:59:11.663 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-09-17T15:59:11.664 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:11.665 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 7.0 M 2024-09-17T15:59:11.665 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2024-09-17T15:59:11.665 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:11.673 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-17T15:59:11.676 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:11.676 INFO:teuthology.orchestra.run.smithi050.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-17T15:59:11.750 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:11.762 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:11.762 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:11.869 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:11.871 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:11.924 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 13/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 1/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-17T15:59:12.001 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-17T15:59:12.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-17T15:59:12.014 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:59:12.082 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:59:12.107 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:12.186 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:12.234 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-17T15:59:12.284 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:59:12.309 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T15:59:12.334 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 640 kB/s | 154 kB 00:00 2024-09-17T15:59:12.337 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-17T15:59:12.337 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T15:59:12.338 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T15:59:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T15:59:12.339 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:12.339 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:12.345 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:59:12.360 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-17T15:59:12.401 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:59:12.460 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-17T15:59:12.461 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:12.462 INFO:teuthology.orchestra.run.smithi006.stdout:Total 8.8 MB/s | 7.0 MB 00:00 2024-09-17T15:59:12.535 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:12.548 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:12.548 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:12.610 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-17T15:59:12.654 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:12.654 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:59:12.812 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:59:12.966 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:59:12.976 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:13.056 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:59:13.093 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:59:13.094 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:13.094 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:13.131 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:59:13.149 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:59:13.162 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:13.187 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:28 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:13.196 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:59:13.321 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:13.322 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:13.322 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:59:13.322 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:13.322 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 154 k 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-17T15:59:13.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:13.324 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-17T15:59:13.324 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-17T15:59:13.324 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:13.357 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-09-17T15:59:13.722 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-09-17T15:59:13.755 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:59:13.772 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-17T15:59:13.797 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-17T15:59:13.814 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 4.4 MB/s | 74 kB 00:00 2024-09-17T15:59:13.859 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:59:13.898 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-09-17T15:59:13.923 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): ceph-mgr-cephadm-19.3.0-4975.g1889a165.e 467 kB/s | 154 kB 00:00 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:59:13.924 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:13.925 INFO:teuthology.orchestra.run.smithi181.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-17T15:59:13.935 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:59:13.939 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:30 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:13.957 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:13.995 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:14.007 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:14.007 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:14.074 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-09-17T15:59:14.075 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:14.076 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 979 k 2024-09-17T15:59:14.076 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.9 M 2024-09-17T15:59:14.076 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:14.107 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:14.107 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:59:14.205 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:59:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:59:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:59:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:59:14.206 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:14.206 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:14.320 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 681 kB/s | 80 kB 00:00 2024-09-17T15:59:14.413 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:14.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-09-17T15:59:14.569 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-17T15:59:14.624 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-17T15:59:14.727 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:14.727 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:59:14.727 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:59:14.727 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:59:14.728 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:59:14.728 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:59:14.753 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 1.6 MB/s | 899 kB 00:00 2024-09-17T15:59:14.754 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:14.754 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.4 MB/s | 979 kB 00:00 2024-09-17T15:59:14.766 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:14.788 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:14.788 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:14.872 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:14.872 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:59:15.014 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:15.015 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:15.047 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:15.058 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:15.140 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-17T15:59:15.186 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:15.187 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-09-17T15:59:15.188 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:15.189 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 979 k 2024-09-17T15:59:15.189 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.9 M 2024-09-17T15:59:15.189 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:15.207 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:59:15.235 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-17T15:59:15.286 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:15.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2024-09-17T15:59:15.303 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-17T15:59:15.329 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:15.467 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 539 kB/s | 80 kB 00:00 2024-09-17T15:59:15.583 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 3.3 MB/s | 899 kB 00:00 2024-09-17T15:59:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:15.584 INFO:teuthology.orchestra.run.smithi050.stdout:Total 2.4 MB/s | 979 kB 00:00 2024-09-17T15:59:15.596 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:15.618 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:15.618 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:15.705 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:15.705 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:15.893 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:15.910 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:15.966 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:15.966 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:59:16.032 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:16.033 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-09-17T15:59:16.034 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:16.034 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 979 k 2024-09-17T15:59:16.034 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.9 M 2024-09-17T15:59:16.034 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:16.048 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 6/6 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 1/6 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-17T15:59:16.107 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-17T15:59:16.127 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:16.263 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:59:16.263 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:16.263 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:16.263 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:59:16.264 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:16.264 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:16.366 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:16.560 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-09-17T15:59:16.561 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 1.8 MB/s | 899 kB 00:00 2024-09-17T15:59:16.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-17T15:59:16.695 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-17T15:59:16.696 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:16.696 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.4 MB/s | 979 kB 00:00 2024-09-17T15:59:16.708 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:16.731 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:16.731 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:16.819 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:16.820 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:16.827 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:16.827 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:59:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:17.087 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:17.162 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:34 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:17.185 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:59:17.216 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:17.256 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:17.301 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:17.302 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:17.302 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 285 k 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.3 M 2024-09-17T15:59:17.303 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:17.349 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 899 k 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:17.350 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:17.351 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-17T15:59:17.351 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:17.351 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 979 k 2024-09-17T15:59:17.351 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.9 M 2024-09-17T15:59:17.352 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:17.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-09-17T15:59:17.517 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:17.517 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:17.518 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:17.518 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:17.526 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:17.526 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:17.552 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:17.552 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:17.608 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 624 kB/s | 80 kB 00:00 2024-09-17T15:59:17.780 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:17.801 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.3.0-4975.g1889a165.el9.x86_ 2.7 MB/s | 899 kB 00:00 2024-09-17T15:59:17.801 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:17.801 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.1 MB/s | 979 kB 00:00 2024-09-17T15:59:17.813 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:17.835 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:17.835 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:17.843 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:17.859 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:17.859 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:17.860 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:59:17.860 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:17.912 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:59:17.917 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:17.917 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:17.979 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:18.098 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:18.120 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:18.120 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:18.120 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 285 k 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.3 M 2024-09-17T15:59:18.121 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:59:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:18.188 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:18.188 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:59:18.189 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:18.189 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:18.253 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-17T15:59:18.324 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:18.346 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-17T15:59:18.347 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:18.347 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-17T15:59:18.347 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:18.357 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:18.357 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:18.385 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:18.386 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:18.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2024-09-17T15:59:18.617 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:18.690 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:18.707 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:18.708 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:18.708 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:59:18.708 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:18.877 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:19.040 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2/2 2024-09-17T15:59:19.040 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/2 2024-09-17T15:59:19.071 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:19.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-09-17T15:59:19.212 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 285 k 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.3 M 2024-09-17T15:59:19.213 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:19.299 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:19.433 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:19.434 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:19.434 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:19.434 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:19.445 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:19.446 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:19.475 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:19.475 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:19.604 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-17T15:59:19.709 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:19.785 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:19.787 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:36 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:19.802 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:19.803 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:59:19.803 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:19.809 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:19.921 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:59:19.922 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 126 k 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 456 k 2024-09-17T15:59:19.923 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:20.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-09-17T15:59:20.113 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 666 kB/s | 126 kB 00:00 2024-09-17T15:59:20.114 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:20.114 INFO:teuthology.orchestra.run.smithi169.stdout:Total 661 kB/s | 126 kB 00:00 2024-09-17T15:59:20.114 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:20.120 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:20.120 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:20.184 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:20.184 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:20.196 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:20.333 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:20.334 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:20.334 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T15:59:20.334 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:20.334 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-4975.g1889a165.el9 ceph-noarch 285 k 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 285 k 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-17T15:59:20.335 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:20.339 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:20.436 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:20.547 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.3.0-4975.g1889a165.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:20.548 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:20.548 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-17T15:59:20.548 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:20.556 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:20.556 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:20.588 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:20.588 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:20.707 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:20.808 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:20.845 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:20.846 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:20.846 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-17T15:59:20.846 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:20.846 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:20.846 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 126 k 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-09-17T15:59:20.847 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:20.873 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:20.888 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:20.890 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:20.890 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T15:59:20.890 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:20.896 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:20.896 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:20.896 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:20.897 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:20.897 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:20.897 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:21.059 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 597 kB/s | 126 kB 00:00 2024-09-17T15:59:21.060 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:21.060 INFO:teuthology.orchestra.run.smithi050.stdout:Total 593 kB/s | 126 kB 00:00 2024-09-17T15:59:21.060 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:21.066 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:21.066 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:21.133 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:21.134 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:21.163 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:21.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2024-09-17T15:59:21.303 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:21.410 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:21.457 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-09-17T15:59:21.775 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:38 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:21.848 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:21.884 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 1/1 2024-09-17T15:59:21.884 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:21.884 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:21.885 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-4975.g1889a165.el9.noarch 2024-09-17T15:59:21.885 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:21.885 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:21.914 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:21.914 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:21.914 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T15:59:21.914 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:21.915 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:21.916 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 126 k 2024-09-17T15:59:21.916 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 456 k 2024-09-17T15:59:21.916 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:22.053 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:22.086 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:22.130 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 590 kB/s | 126 kB 00:00 2024-09-17T15:59:22.131 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:22.131 INFO:teuthology.orchestra.run.smithi006.stdout:Total 586 kB/s | 126 kB 00:00 2024-09-17T15:59:22.131 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:22.137 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:22.137 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:22.146 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:22.192 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:22.193 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:22.193 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:22.205 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:22.205 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:22.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-17T15:59:22.261 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-09-17T15:59:22.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-09-17T15:59:22.363 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:22.465 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:22.807 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:22.861 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:39 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:22.940 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:22.941 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:22.941 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-17T15:59:22.941 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:22.941 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 126 k 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 126 k 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-09-17T15:59:22.942 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:22.969 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:22.978 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:23.007 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:23.007 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 32 k 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 142 k 2024-09-17T15:59:23.008 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:23.071 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:23.121 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:23.123 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:23.123 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:23.159 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 211 kB/s | 32 kB 00:00 2024-09-17T15:59:23.160 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.3.0-4975.g1889a165.el9.x86_64 582 kB/s | 126 kB 00:00 2024-09-17T15:59:23.160 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:23.161 INFO:teuthology.orchestra.run.smithi169.stdout:Total 208 kB/s | 32 kB 00:00 2024-09-17T15:59:23.161 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:23.161 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:23.161 INFO:teuthology.orchestra.run.smithi181.stdout:Total 578 kB/s | 126 kB 00:00 2024-09-17T15:59:23.161 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:23.164 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:23.164 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:23.166 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:23.166 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:23.187 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:23.188 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:23.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:23.223 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:23.229 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:23.230 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:23.316 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:23.395 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:23.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2024-09-17T15:59:23.475 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:23.491 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:23.805 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:23.865 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:23.934 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:23.942 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:23.943 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:23.944 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:23.944 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 32 k 2024-09-17T15:59:23.944 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 142 k 2024-09-17T15:59:23.944 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:24.081 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:24.089 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 219 kB/s | 32 kB 00:00 2024-09-17T15:59:24.089 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:24.090 INFO:teuthology.orchestra.run.smithi050.stdout:Total 217 kB/s | 32 kB 00:00 2024-09-17T15:59:24.090 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:24.093 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:24.094 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:24.110 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:24.110 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:24.111 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:24.111 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:24.111 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:24.111 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:24.113 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:24.113 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:24.175 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:24.189 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:24.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:24.190 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:24.190 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:24.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:24.190 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:24.194 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:24.220 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:24.222 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:24.222 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:24.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2024-09-17T15:59:24.307 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-09-17T15:59:24.307 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:24.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-17T15:59:24.672 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:24.889 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:24.897 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:24.901 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:24.979 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:25.023 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:25.024 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:25.024 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:25.036 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:25.037 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:25.038 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:25.038 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:25.038 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 32 k 2024-09-17T15:59:25.038 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 142 k 2024-09-17T15:59:25.038 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:25.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-09-17T15:59:25.102 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-09-17T15:59:25.110 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:25.185 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 217 kB/s | 32 kB 00:00 2024-09-17T15:59:25.185 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:25.185 INFO:teuthology.orchestra.run.smithi006.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-17T15:59:25.185 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:25.206 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:25.209 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:25.209 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:25.251 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:25.252 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:25.253 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:25.322 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-17T15:59:25.419 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:25.535 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:25.676 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:25.678 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:25.768 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:25.770 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:25.811 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:25.812 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:25.812 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:25.815 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:25.817 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:25.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-09-17T15:59:25.894 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:25.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-09-17T15:59:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:26.027 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-4975.g1889a165.el9 ceph 32 k 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:26.028 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-09-17T15:59:26.029 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-09-17T15:59:26.029 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:26.160 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:26.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:26.161 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:26.161 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:26.161 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:26.161 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:26.181 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.3.0-4975.g1889a165.el9.x86_6 208 kB/s | 32 kB 00:00 2024-09-17T15:59:26.181 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:26.182 INFO:teuthology.orchestra.run.smithi181.stdout:Total 206 kB/s | 32 kB 00:00 2024-09-17T15:59:26.182 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:26.189 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:26.189 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:26.208 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:26.208 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:26.295 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:26.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2024-09-17T15:59:26.412 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:26.469 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:26.503 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:26.558 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:26.596 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:26.601 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:26.602 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:26.602 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:26.641 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:26.642 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:26.642 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:26.671 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-09-17T15:59:26.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-09-17T15:59:26.824 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:26.936 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:27.029 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:27.055 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:27.074 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:27.075 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:27.076 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:27.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2024-09-17T15:59:27.246 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:27.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-17T15:59:27.321 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:27.336 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:27.379 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:27.380 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:27.380 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:27.413 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:27.454 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-09-17T15:59:27.459 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:27.460 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:27.460 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:27.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-09-17T15:59:27.744 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:27.837 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:27.849 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:42 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:27.882 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:27.883 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:27.883 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:27.938 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:27.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2024-09-17T15:59:27.982 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:27.983 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:27.983 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:28.034 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:28.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-17T15:59:28.124 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:28.146 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:28.167 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:28.168 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:28.168 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:28.234 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-09-17T15:59:28.239 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:28.285 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:28.286 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:28.286 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:28.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-09-17T15:59:28.555 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:28.646 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:28.647 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:28.694 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:28.695 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:28.695 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:28.735 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:28.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2024-09-17T15:59:28.779 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:28.780 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:28.780 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:28.815 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:28.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-17T15:59:28.906 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:28.950 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:28.951 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-09-17T15:59:28.951 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:28.968 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:29.026 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-09-17T15:59:29.061 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:29.107 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:29.108 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:29.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-09-17T15:59:29.371 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:29.443 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:44 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:29.464 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:29.511 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:29.512 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:29.512 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:29.576 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:29.577 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:29.577 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:29.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2024-09-17T15:59:29.608 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:29.650 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-17T15:59:29.740 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:59:29.741 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 85 k 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 231 k 2024-09-17T15:59:29.742 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:29.795 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:29.887 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:29.933 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-17T15:59:29.934 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:30.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-09-17T15:59:30.087 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 247 kB/s | 85 kB 00:00 2024-09-17T15:59:30.088 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:30.088 INFO:teuthology.orchestra.run.smithi169.stdout:Total 246 kB/s | 85 kB 00:00 2024-09-17T15:59:30.088 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:30.095 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:30.095 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:30.160 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:30.160 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:30.185 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:30.229 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:45 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:30.278 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:30.300 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:30.320 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:30.325 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:30.326 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:30.326 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:30.363 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:30.364 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:30.365 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:30.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2024-09-17T15:59:30.393 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:30.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-17T15:59:30.614 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:50 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:30.753 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:30.754 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 85 k 2024-09-17T15:59:30.755 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 231 k 2024-09-17T15:59:30.755 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:30.787 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:30.996 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:31.017 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:31.042 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 296 kB/s | 85 kB 00:00 2024-09-17T15:59:31.042 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:31.043 INFO:teuthology.orchestra.run.smithi050.stdout:Total 295 kB/s | 85 kB 00:00 2024-09-17T15:59:31.043 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:31.051 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:31.051 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:31.060 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:31.061 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:31.061 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:31.061 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:31.061 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:31.061 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:31.091 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:31.105 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:31.117 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:31.118 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:31.138 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:31.139 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-09-17T15:59:31.139 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:31.150 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:31.151 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:31.151 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:31.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2024-09-17T15:59:31.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-17T15:59:31.268 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:31.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-09-17T15:59:31.385 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:31.743 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:31.800 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:46 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:31.815 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:31.889 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.3.0-4975.g1889a165.el9.x86_64 is already installed. 2024-09-17T15:59:31.913 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:48 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:31.932 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:31.934 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-17T15:59:31.934 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:31.953 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:31.955 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 85 k 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 231 k 2024-09-17T15:59:31.956 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:32.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-17T15:59:32.050 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:32.051 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:32.052 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:32.053 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:32.053 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.2 M 2024-09-17T15:59:32.053 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 14 M 2024-09-17T15:59:32.053 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:32.147 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 445 kB/s | 85 kB 00:00 2024-09-17T15:59:32.147 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:32.147 INFO:teuthology.orchestra.run.smithi006.stdout:Total 442 kB/s | 85 kB 00:00 2024-09-17T15:59:32.147 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:32.155 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:32.156 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:32.222 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:32.222 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:32.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-09-17T15:59:32.377 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:32.391 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:59:32.391 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:32.392 INFO:teuthology.orchestra.run.smithi169.stdout:Total 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:59:32.392 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:32.401 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:32.401 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:32.483 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:32.483 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:32.488 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:32.582 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:47 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:32.715 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-4975.g1889a165.el9 ceph 85 k 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:32.716 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-09-17T15:59:32.717 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-09-17T15:59:32.717 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:32.865 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:32.924 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:32.926 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.3.0-4975.g1889a165.el9.x86_64.rpm 407 kB/s | 85 kB 00:00 2024-09-17T15:59:32.926 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:32.926 INFO:teuthology.orchestra.run.smithi181.stdout:Total 405 kB/s | 85 kB 00:00 2024-09-17T15:59:32.927 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:32.933 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:33.008 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:33.009 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:33.010 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:33.123 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:59:33.124 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.2 M 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 14 M 2024-09-17T15:59:33.125 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:33.138 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:33.139 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:33.139 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:33.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:33.139 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:33.142 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:33.248 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-09-17T15:59:33.470 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.3 MB/s | 3.2 MB 00:00 2024-09-17T15:59:33.471 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:33.471 INFO:teuthology.orchestra.run.smithi050.stdout:Total 9.2 MB/s | 3.2 MB 00:00 2024-09-17T15:59:33.471 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:33.481 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:33.481 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:33.567 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:33.567 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.916 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:33.916 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:33.916 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:33.916 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:33.917 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:34.025 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:34.034 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:34.102 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:34.103 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:34.103 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:34.103 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:34.103 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:34.103 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:34.104 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:34.131 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:34.132 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:34.132 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:59:34.132 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:34.132 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:34.132 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:34.163 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.2 M 2024-09-17T15:59:34.164 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 14 M 2024-09-17T15:59:34.165 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:34.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-17T15:59:34.335 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-09-17T15:59:34.516 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.1 MB/s | 3.2 MB 00:00 2024-09-17T15:59:34.517 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:34.517 INFO:teuthology.orchestra.run.smithi006.stdout:Total 9.0 MB/s | 3.2 MB 00:00 2024-09-17T15:59:34.517 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:34.527 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:34.528 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:34.614 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:34.614 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:34.785 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:49 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:34.929 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:34.929 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:34.929 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-17T15:59:34.929 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:34.929 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-4975.g1889a165.el9 ceph 3.2 M 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:34.930 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.2 M 2024-09-17T15:59:34.931 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 14 M 2024-09-17T15:59:34.931 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:34.935 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:51 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:35.069 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-09-17T15:59:35.070 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:35.070 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:35.070 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:35.070 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 170 k 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 490 k 2024-09-17T15:59:35.071 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-09-17T15:59:35.083 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:35.142 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.168 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.168 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:35.169 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:59:35.169 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:35.169 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:35.169 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:35.261 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 897 kB/s | 170 kB 00:00 2024-09-17T15:59:35.262 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:35.262 INFO:teuthology.orchestra.run.smithi169.stdout:Total 891 kB/s | 170 kB 00:00 2024-09-17T15:59:35.262 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-09-17T15:59:35.269 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-09-17T15:59:35.269 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-09-17T15:59:35.271 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.3.0-4975.g1889a165.el9.x86_64.rpm 9.4 MB/s | 3.2 MB 00:00 2024-09-17T15:59:35.272 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:35.272 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.3 MB/s | 3.2 MB 00:00 2024-09-17T15:59:35.273 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:35.282 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:35.282 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:35.323 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:35.334 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-09-17T15:59:35.334 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-09-17T15:59:35.368 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:35.369 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:35.484 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-09-17T15:59:35.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-09-17T15:59:35.584 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.811 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:35.880 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T15:59:35.906 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:36.044 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:36.168 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:56 ago on Tue 17 Sep 2024 03:57:40 PM UTC. 2024-09-17T15:59:36.310 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-17T15:59:36.310 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 170 k 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 490 k 2024-09-17T15:59:36.311 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-17T15:59:36.317 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:36.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:36.318 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-09-17T15:59:36.318 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:36.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:36.318 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-09-17T15:59:36.336 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:36.336 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:36.336 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:36.336 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:36.336 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:36.337 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:36.524 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 803 kB/s | 170 kB 00:00 2024-09-17T15:59:36.524 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:36.524 INFO:teuthology.orchestra.run.smithi050.stdout:Total 797 kB/s | 170 kB 00:00 2024-09-17T15:59:36.525 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-17T15:59:36.534 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-17T15:59:36.534 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-17T15:59:36.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2024-09-17T15:59:36.599 DEBUG:teuthology.parallel:result is None 2024-09-17T15:59:36.601 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-17T15:59:36.602 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-17T15:59:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-17T15:59:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:37.021 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:37.021 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:37.022 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:37.022 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:37.022 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:37.022 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:37.159 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:54 ago on Tue 17 Sep 2024 03:57:43 PM UTC. 2024-09-17T15:59:37.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-17T15:59:37.297 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-09-17T15:59:37.298 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-09-17T15:59:37.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:37.299 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 170 k 2024-09-17T15:59:37.299 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 490 k 2024-09-17T15:59:37.299 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-09-17T15:59:37.508 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:37.529 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 742 kB/s | 170 kB 00:00 2024-09-17T15:59:37.529 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:37.529 INFO:teuthology.orchestra.run.smithi006.stdout:Total 737 kB/s | 170 kB 00:00 2024-09-17T15:59:37.530 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-09-17T15:59:37.538 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-09-17T15:59:37.538 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-09-17T15:59:37.606 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-09-17T15:59:37.606 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-09-17T15:59:37.765 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:37.766 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:37.766 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-17T15:59:37.766 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:37.766 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:37.766 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-17T15:59:37.768 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-09-17T15:59:37.870 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:37.872 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Tue 17 Sep 2024 03:57:45 PM UTC. 2024-09-17T15:59:38.003 DEBUG:teuthology.parallel:result is None 2024-09-17T15:59:38.005 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T15:59:38.006 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-4975.g1889a165.el9 ceph 170 k 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 490 k 2024-09-17T15:59:38.007 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T15:59:38.216 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.3.0-4975.g1889a165.el9.x86_64.rpm 816 kB/s | 170 kB 00:00 2024-09-17T15:59:38.217 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T15:59:38.217 INFO:teuthology.orchestra.run.smithi181.stdout:Total 810 kB/s | 170 kB 00:00 2024-09-17T15:59:38.217 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T15:59:38.225 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T15:59:38.225 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T15:59:38.289 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T15:59:38.289 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T15:59:38.289 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:38.441 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T15:59:38.526 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:38.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:38.526 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-09-17T15:59:38.526 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:38.526 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:38.527 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-09-17T15:59:38.548 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:38.783 DEBUG:teuthology.parallel:result is None 2024-09-17T15:59:38.958 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 1/1 2024-09-17T15:59:39.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:39.213 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T15:59:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-4975.g1889a165.el9.x86_64 2024-09-17T15:59:39.214 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:39.214 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T15:59:39.463 DEBUG:teuthology.parallel:result is None 2024-09-17T15:59:39.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:39.620 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:59:39.646 INFO:teuthology.orchestra.run.smithi006.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:59:39.647 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:59:39.647 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:59:39.649 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:39.797 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:59:39.824 INFO:teuthology.orchestra.run.smithi050.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:59:39.825 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:59:39.825 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:59:39.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:39.976 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:59:40.002 INFO:teuthology.orchestra.run.smithi169.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:59:40.002 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:59:40.003 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:59:40.004 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:59:40.150 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T15:59:40.173 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-4975.g1889a165.el9 2024-09-17T15:59:40.173 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975.g1889a165.el9 2024-09-17T15:59:40.173 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975.g1889a165 is installed. 2024-09-17T15:59:40.175 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T15:59:40.175 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:40.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:59:40.207 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:40.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:59:40.239 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:40.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:59:40.270 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:40.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:59:40.296 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T15:59:40.297 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:40.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:59:40.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:59:40.398 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:40.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:59:40.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:59:40.501 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:40.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:59:40.531 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:59:40.601 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:40.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:59:40.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:59:40.692 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T15:59:40.693 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:40.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:59:40.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:59:40.793 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:40.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:59:40.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:59:40.895 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:40.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:59:40.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:59:40.994 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:40.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:59:41.021 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:59:41.087 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T15:59:41.087 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:41.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:59:41.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:59:41.189 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:41.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:59:41.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:59:41.291 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:41.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:59:41.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:59:41.390 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:41.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:59:41.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:59:41.481 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T15:59:41.591 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T15:59:41.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2024-09-17T15:59:41.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-09-17T15:59:41.598 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-09-17T15:59:41.600 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-17T15:59:41.627 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T15:59:41.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:59:41.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:59:41.669 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:59:41.672 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:59:41.698 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T15:59:41.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:59:41.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:59:41.741 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:59:41.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:59:41.783 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T15:59:41.783 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T15:59:41.783 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015', 'branch': 'wip-bharath10-testing-2024-09-17-1348', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7909015', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 3421, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath10-testing-2024-09-17-1348', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC4xLWJ353EqAA8CBGLNR4urZmSC/KeKWZSkSfxUnZ55qfKt4C4BIIncLlbs6PBP4Lj5d1mrdMou59vTnPekfnU=', 'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFIDjkEfvEI08ayJYXBLdWl/N3OpN6iqDJeeZ48TfT+0F3G4BtcU+fvbuXdIgO9NMoG5NEVNLDyiKykRZbTNAbU=', 'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE1xpK4yttHOVDgw5AKw8mOQPv4I4SciXhDvIuBWKq/jOGje+IqmqmDC3JlBNemPoCEecSDE+o6wPQ78lStIcl4=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOWyqsqxRsfUAs1kNo0Fx71ryEubVxTm7cBWYH8rSWgaB+D6IZHUWWd5BRPx9NGo6Xrm8FCE95rJIf1CvF6VpnY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'cluster': 'ceph'}}, {'thrashosds': {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_15:01:33', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T15:59:41.784 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:59:41.807 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:59:41.811 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:59:41.814 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:59:41.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:59:41.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:59:41.870 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:59:41.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:59:41.900 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:41.900 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:59:41.950 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:59:41.951 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-09-17T15:59:42.008 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 15:59:35.422920784 +0000 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 15:57:01.669757079 +0000 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 15:57:01.669757079 +0000 2024-09-17T15:59:42.009 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 15:57:01.669757079 +0000 2024-09-17T15:59:42.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:59:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T15:59:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T15:59:42.076 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000286701 s, 1.8 MB/s 2024-09-17T15:59:42.077 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:59:42.136 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 15:59:35.423920773 +0000 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 15:57:01.999753386 +0000 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 15:57:01.999753386 +0000 2024-09-17T15:59:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 15:57:01.999753386 +0000 2024-09-17T15:59:42.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:59:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T15:59:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T15:59:42.261 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000201136 s, 2.5 MB/s 2024-09-17T15:59:42.262 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:59:42.321 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-09-17T15:59:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 15:59:35.423920773 +0000 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 15:57:02.333749648 +0000 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 15:57:02.333749648 +0000 2024-09-17T15:59:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 15:57:02.333749648 +0000 2024-09-17T15:59:42.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:59:42.446 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T15:59:42.446 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T15:59:42.446 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000198689 s, 2.6 MB/s 2024-09-17T15:59:42.447 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:59:42.505 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-09-17T15:59:42.562 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:59:42.562 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.562 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T15:59:42.562 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.562 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.563 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-09-17 15:59:35.424920762 +0000 2024-09-17T15:59:42.563 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-09-17 15:57:02.667745909 +0000 2024-09-17T15:59:42.563 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-09-17 15:57:02.667745909 +0000 2024-09-17T15:59:42.563 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-09-17 15:57:02.667745909 +0000 2024-09-17T15:59:42.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:59:42.628 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-09-17T15:59:42.629 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-09-17T15:59:42.629 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00020315 s, 2.5 MB/s 2024-09-17T15:59:42.630 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:59:42.689 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:42.689 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:42.689 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:59:42.710 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:59:42.710 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-09-17T15:59:42.769 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-17 15:59:34.403104436 +0000 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-17 15:57:02.839624761 +0000 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-17 15:57:02.839624761 +0000 2024-09-17T15:59:42.770 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-17 15:57:02.839624761 +0000 2024-09-17T15:59:42.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:59:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-17T15:59:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-17T15:59:42.838 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000122005 s, 4.2 MB/s 2024-09-17T15:59:42.840 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:59:42.900 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-17 15:59:34.404104409 +0000 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-17 15:57:03.155616214 +0000 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-17 15:57:03.155616214 +0000 2024-09-17T15:59:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-17 15:57:03.155616214 +0000 2024-09-17T15:59:42.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:59:43.028 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-17T15:59:43.028 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-17T15:59:43.028 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.2392e-05 s, 5.5 MB/s 2024-09-17T15:59:43.029 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:59:43.089 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-09-17T15:59:43.148 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:59:43.148 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T15:59:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-17 15:59:34.404104409 +0000 2024-09-17T15:59:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-17 15:57:03.493607071 +0000 2024-09-17T15:59:43.150 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-17 15:57:03.493607071 +0000 2024-09-17T15:59:43.150 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-17 15:57:03.493607071 +0000 2024-09-17T15:59:43.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:59:43.217 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-17T15:59:43.218 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-17T15:59:43.218 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000117682 s, 4.4 MB/s 2024-09-17T15:59:43.219 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:59:43.278 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-17 15:59:34.404104409 +0000 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-17 15:57:03.820598226 +0000 2024-09-17T15:59:43.338 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-17 15:57:03.820598226 +0000 2024-09-17T15:59:43.339 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-17 15:57:03.820598226 +0000 2024-09-17T15:59:43.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:59:43.406 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-17T15:59:43.406 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-17T15:59:43.406 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000103165 s, 5.0 MB/s 2024-09-17T15:59:43.407 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:59:43.467 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:43.467 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:43.467 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:59:43.486 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:59:43.486 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-09-17T15:59:43.545 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-17 15:59:33.274615738 +0000 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-17 15:56:59.085041257 +0000 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-17 15:56:59.085041257 +0000 2024-09-17T15:59:43.546 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-17 15:56:59.085041257 +0000 2024-09-17T15:59:43.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:59:43.614 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-17T15:59:43.614 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-17T15:59:43.614 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000228755 s, 2.2 MB/s 2024-09-17T15:59:43.616 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:59:43.675 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-09-17T15:59:43.732 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-17 15:59:33.275615718 +0000 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-17 15:56:59.414034660 +0000 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-17 15:56:59.414034660 +0000 2024-09-17T15:59:43.733 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-17 15:56:59.414034660 +0000 2024-09-17T15:59:43.734 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:59:43.799 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-17T15:59:43.799 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-17T15:59:43.799 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000198158 s, 2.6 MB/s 2024-09-17T15:59:43.800 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:59:43.858 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-17 15:59:33.275615718 +0000 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-17 15:56:59.738028162 +0000 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-17 15:56:59.738028162 +0000 2024-09-17T15:59:43.915 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-17 15:56:59.738028162 +0000 2024-09-17T15:59:43.916 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:59:43.981 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-17T15:59:43.981 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-17T15:59:43.981 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000239397 s, 2.1 MB/s 2024-09-17T15:59:43.982 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:59:44.039 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-09-17 15:59:33.275615718 +0000 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-09-17 15:57:00.071021484 +0000 2024-09-17T15:59:44.095 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-09-17 15:57:00.071021484 +0000 2024-09-17T15:59:44.096 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-09-17 15:57:00.071021484 +0000 2024-09-17T15:59:44.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:59:44.159 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-09-17T15:59:44.159 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-09-17T15:59:44.159 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000209538 s, 2.4 MB/s 2024-09-17T15:59:44.161 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:59:44.217 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:44.218 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:44.218 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:59:44.237 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:59:44.237 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 15:59:36.160145828 +0000 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 15:56:59.063928661 +0000 2024-09-17T15:59:44.296 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 15:56:59.063928661 +0000 2024-09-17T15:59:44.297 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 15:56:59.063928661 +0000 2024-09-17T15:59:44.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:59:44.364 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T15:59:44.364 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T15:59:44.364 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000224561 s, 2.3 MB/s 2024-09-17T15:59:44.366 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:59:44.424 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-17T15:59:44.482 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:59:44.482 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:44.482 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-17T15:59:44.482 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:44.483 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:44.483 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 15:59:36.160145828 +0000 2024-09-17T15:59:44.483 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 15:56:59.389921870 +0000 2024-09-17T15:59:44.483 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 15:56:59.389921870 +0000 2024-09-17T15:59:44.483 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 15:56:59.389921870 +0000 2024-09-17T15:59:44.483 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:59:44.549 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T15:59:44.549 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T15:59:44.549 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000203723 s, 2.5 MB/s 2024-09-17T15:59:44.551 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:59:44.609 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-17T15:59:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:59:44.666 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 15:59:36.161145807 +0000 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 15:56:59.704915312 +0000 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 15:56:59.704915312 +0000 2024-09-17T15:59:44.667 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 15:56:59.704915312 +0000 2024-09-17T15:59:44.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:59:44.732 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T15:59:44.732 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T15:59:44.732 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000191055 s, 2.7 MB/s 2024-09-17T15:59:44.733 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:59:44.791 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-17T15:59:44.848 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:59:44.848 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:59:44.848 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T15:59:44.848 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:59:44.848 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T15:59:44.849 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 15:59:36.161145807 +0000 2024-09-17T15:59:44.849 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 15:57:00.037908379 +0000 2024-09-17T15:59:44.849 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 15:57:00.037908379 +0000 2024-09-17T15:59:44.849 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 15:57:00.037908379 +0000 2024-09-17T15:59:44.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:59:44.913 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T15:59:44.913 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T15:59:44.913 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000180351 s, 2.8 MB/s 2024-09-17T15:59:44.914 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:59:44.972 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:44.972 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-17T15:59:44.972 INFO:tasks.ceph:Generating config... 2024-09-17T15:59:44.973 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-17T15:59:44.973 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T15:59:44.973 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-17T15:59:44.974 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-17T15:59:44.975 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-17T15:59:44.976 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-17T15:59:44.976 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T15:59:44.976 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T15:59:44.976 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T15:59:44.977 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T15:59:44.977 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T15:59:44.977 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-17T15:59:44.977 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T15:59:44.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T15:59:45.042 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T15:59:45.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T15:59:45.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:59:45.122 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.b', '172.21.15.50'), ('mon.c', '172.21.15.169')] 2024-09-17T15:59:45.122 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,172.21.15.50,172.21.15.169', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-17T15:59:45.123 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:45.123 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:59:45.178 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --add b 172.21.15.50 --add c 172.21.15.169 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:45.267 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2024-09-17T15:59:45.266543+0000 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-17T15:59:45.266543+0000 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 17 (quincy) 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.b 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.c 2024-09-17T15:59:45.268 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T15:59:45.270 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:59:45.285 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4345ca04-2628-4bef-a537-6837268b96c9... 2024-09-17T15:59:45.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:59:45.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:59:45.330 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:59:45.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:59:45.370 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-09-17T15:59:45.370 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.371 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2024-09-17T15:59:45.372 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2024-09-17T15:59:45.373 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,172.21.15.50,172.21.15.169" 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 1 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = osd 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject internal delays = 0.002 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2024-09-17T15:59:45.374 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 120 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug trim objects = true 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.375 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2024-09-17T15:59:45.376 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = lz4 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore write v2 random = True 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2024-09-17T15:59:45.377 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd blocked scrub grace period = 3600 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug reject backfill probability = 0.3 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd max backfills = 3 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd max markdown count = 1000 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub min interval = 60 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: osd shutdown pgref assert = True 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.378 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2024-09-17T15:59:45.379 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:59:45.380 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2024-09-17T15:59:45.381 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2024-09-17T15:59:45.382 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-09-17T15:59:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-09-17T15:59:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:59:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.6,172.21.15.50,172.21.15.169" 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 1 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay max = 1 2024-09-17T15:59:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay probability = 0.005 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject delay type = osd 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject internal delays = 0.002 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 2500 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 120 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:59:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug trim objects = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-09-17T15:59:45.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: debug monc = 20 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression algorithm = lz4 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore compression mode = aggressive 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore write v2 random = True 2024-09-17T15:59:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd blocked scrub grace period = 3600 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug reject backfill probability = 0.3 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:59:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: osd max backfills = 3 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd max markdown count = 1000 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub min interval = 60 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: osd shutdown pgref assert = True 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-17T15:59:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2024-09-17T15:59:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:59:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:59:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = "172.21.15.6,172.21.15.50,172.21.15.169" 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon client directed command retry = 5 2024-09-17T15:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout: mon election default strategy = 1 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject delay max = 1 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject delay probability = 0.005 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject delay type = osd 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject internal delays = 0.002 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: ms inject socket failures = 2500 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 120 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:59:45.398 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.399 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore block size = 96636764160 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression algorithm = lz4 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore compression mode = aggressive 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore fsck on mount = True 2024-09-17T15:59:45.400 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore write v2 random = True 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: bluestore zero block detection = True 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluefs = 20 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: debug bluestore = 20 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: debug rocksdb = 10 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd full ratio = 0.9 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: osd blocked scrub grace period = 3600 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug reject backfill probability = 0.3 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify cached snaps = True 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify missing on start = True 2024-09-17T15:59:45.401 INFO:teuthology.orchestra.run.smithi169.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd max backfills = 3 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd max markdown count = 1000 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock override recovery settings = True 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd objectstore = bluestore 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue = debug_random 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd op queue cut off = debug_random 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub min interval = 60 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-09-17T15:59:45.402 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-09-17T15:59:45.403 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.404 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: mon scrub interval = 300 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.b] 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.c] 2024-09-17T15:59:45.405 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T15:59:45.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T15:59:45.407 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.408 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.6,172.21.15.50,172.21.15.169" 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: mon client directed command retry = 5 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 1 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay max = 1 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay probability = 0.005 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject delay type = osd 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject internal delays = 0.002 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: ms inject socket failures = 2500 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-17T15:59:45.409 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 120 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T15:59:45.410 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:59:45.411 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression algorithm = lz4 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression mode = aggressive 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore write v2 random = True 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T15:59:45.412 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd blocked scrub grace period = 3600 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug reject backfill probability = 0.3 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd max backfills = 3 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd max markdown count = 1000 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T15:59:45.413 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub min interval = 60 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T15:59:45.414 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:59:45.415 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T15:59:45.416 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T15:59:45.417 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T15:59:45.419 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T15:59:45.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T15:59:45.514 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T15:59:45.514 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:45.514 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T15:59:45.530 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:45.530 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T15:59:45.587 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:59:45.587 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:45.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:59:45.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:59:45.662 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T15:59:45.663 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:45.718 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T15:59:45.719 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:45.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:59:45.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:59:45.762 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T15:59:45.762 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:45.820 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T15:59:45.820 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:45.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:59:45.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:59:45.862 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T15:59:45.862 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:45.920 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T15:59:45.920 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:45.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:59:45.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:59:45.959 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T15:59:45.959 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:46.016 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T15:59:46.016 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T15:59:46.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T15:59:46.071 INFO:teuthology.orchestra.run.smithi006.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T15:59:46.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:59:46.156 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:59:46.159 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T15:59:46.159 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T15:59:46.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T15:59:46.202 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T15:59:46.214 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T15:59:46.215 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-17T15:59:46.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T15:59:46.241 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:46.241 INFO:tasks.ceph:role: osd.0 2024-09-17T15:59:46.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:59:46.241 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T15:59:46.315 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:59:46.315 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:59:46.316 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:59:46.831 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T15:59:46.833 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T15:59:46.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T15:59:46.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T15:59:47.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T15:59:47.077 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:47.077 INFO:tasks.ceph:role: osd.4 2024-09-17T15:59:47.077 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:59:47.078 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T15:59:47.153 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:59:47.154 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:59:47.614 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T15:59:47.616 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T15:59:47.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-17T15:59:47.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T15:59:47.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-17T15:59:47.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:47.804 INFO:tasks.ceph:role: osd.8 2024-09-17T15:59:47.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:59:47.804 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:59:47.878 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:59:47.879 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:59:47.879 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:59:48.366 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T15:59:48.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T15:59:48.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-17T15:59:48.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-17T15:59:48.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-17T15:59:48.610 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-17T15:59:48.610 INFO:tasks.ceph:role: osd.12 2024-09-17T15:59:48.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T15:59:48.610 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T15:59:48.683 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:59:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:59:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:59:48.684 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:59:49.166 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2024-09-17T15:59:49.169 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2024-09-17T15:59:49.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-17T15:59:49.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-17T15:59:49.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:49.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:49.436+0000 7f4e4a3b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T15:59:49.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:49.436+0000 7f4e4a3b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T15:59:49.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:49.436+0000 7f4e4a3b2640 -1 bdev(0x55557c729800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T15:59:49.440 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:49.436+0000 7f4e4a3b2640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T15:59:52.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T15:59:52.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:52.480+0000 7fce9800f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T15:59:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:52.480+0000 7fce9800f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T15:59:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:52.480+0000 7fce9800f640 -1 bdev(0x559242f05800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T15:59:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:52.480+0000 7fce9800f640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T15:59:55.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T15:59:55.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:55.759+0000 7fa4f4a14640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-17T15:59:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:55.759+0000 7fa4f4a14640 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-17T15:59:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:55.759+0000 7fa4f4a14640 -1 bdev(0x56476695d800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-17T15:59:55.761 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:55.759+0000 7fa4f4a14640 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-17T15:59:58.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-17T15:59:58.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:59:59.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:59.054+0000 7fa17c217640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-17T15:59:59.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:59.054+0000 7fa17c217640 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-17T15:59:59.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:59.054+0000 7fa17c217640 -1 bdev(0x562ace7f1800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-17T15:59:59.056 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T15:59:59.054+0000 7fa17c217640 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-17T16:00:01.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-17T16:00:02.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T16:00:02.052 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:02.052 INFO:tasks.ceph:role: osd.1 2024-09-17T16:00:02.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:00:02.052 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:02.122 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:02.123 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:02.123 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:02.123 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:02.156 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-17T16:00:02.158 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-17T16:00:02.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-17T16:00:02.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T16:00:02.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T16:00:02.371 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:02.371 INFO:tasks.ceph:role: osd.5 2024-09-17T16:00:02.372 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:00:02.372 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:02.437 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:02.438 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:02.438 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:02.438 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:02.469 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-17T16:00:02.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-17T16:00:02.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-17T16:00:02.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T16:00:02.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-17T16:00:02.678 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:02.678 INFO:tasks.ceph:role: osd.9 2024-09-17T16:00:02.678 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:00:02.678 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:02.748 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:02.780 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-17T16:00:02.782 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-17T16:00:02.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-17T16:00:02.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-17T16:00:02.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-17T16:00:02.988 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:02.988 INFO:tasks.ceph:role: osd.13 2024-09-17T16:00:02.988 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:00:02.988 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T16:00:03.053 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:03.053 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:03.053 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:03.053 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:03.054 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:03.086 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-17T16:00:03.088 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-17T16:00:03.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-17T16:00:03.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-17T16:00:03.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:03.326+0000 7f0ed26b4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T16:00:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:03.326+0000 7f0ed26b4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T16:00:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:03.326+0000 7f0ed26b4640 -1 bdev(0x55e378249800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:03.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:03.326+0000 7f0ed26b4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T16:00:06.724 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T16:00:06.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:06.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:06.845+0000 7f77cf42e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T16:00:06.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:06.845+0000 7f77cf42e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T16:00:06.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:06.846+0000 7f77cf42e640 -1 bdev(0x55b570ac7800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:06.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:06.846+0000 7f77cf42e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T16:00:10.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T16:00:10.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:10.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:10.637+0000 7f9448c16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-17T16:00:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:10.637+0000 7f9448c16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-17T16:00:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:10.637+0000 7f9448c16640 -1 bdev(0x563d3d2c3800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:10.637+0000 7f9448c16640 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-17T16:00:14.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-17T16:00:14.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:14.432+0000 7f525112c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-17T16:00:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:14.433+0000 7f525112c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-17T16:00:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:14.433+0000 7f525112c640 -1 bdev(0x555b93735800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:14.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-17T16:00:14.433+0000 7f525112c640 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-17T16:00:17.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-17T16:00:17.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T16:00:17.657 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:17.657 INFO:tasks.ceph:role: osd.2 2024-09-17T16:00:17.657 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:00:17.657 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:17.732 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:17.733 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:17.733 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:17.733 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:18.241 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-09-17T16:00:18.243 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-09-17T16:00:18.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-17T16:00:18.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T16:00:18.397 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-17T16:00:18.462 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:18.462 INFO:tasks.ceph:role: osd.6 2024-09-17T16:00:18.462 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:00:18.462 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:18.534 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:19.007 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-09-17T16:00:19.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-09-17T16:00:19.009 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-17T16:00:19.096 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-17T16:00:19.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-17T16:00:19.190 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:19.190 INFO:tasks.ceph:role: osd.10 2024-09-17T16:00:19.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:00:19.191 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T16:00:19.262 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:19.263 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:19.752 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-09-17T16:00:19.754 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-09-17T16:00:19.755 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-17T16:00:19.903 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-17T16:00:19.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-17T16:00:19.996 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:19.997 INFO:tasks.ceph:role: osd.14 2024-09-17T16:00:19.997 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:00:19.997 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T16:00:20.069 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:20.069 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:20.069 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:20.069 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:20.069 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:20.070 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:20.070 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:20.070 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:20.070 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:20.070 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:20.597 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-09-17T16:00:20.598 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-09-17T16:00:20.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-17T16:00:20.743 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-17T16:00:20.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:20.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:20.869+0000 7fbbe98ed640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T16:00:20.871 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:20.869+0000 7fbbe98ed640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T16:00:20.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:20.869+0000 7fbbe98ed640 -1 bdev(0x556ad24c3800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:20.872 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:20.869+0000 7fbbe98ed640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T16:00:24.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T16:00:24.063 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:24.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:24.150+0000 7f90134ef640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-17T16:00:24.152 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:24.150+0000 7f90134ef640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-17T16:00:24.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:24.150+0000 7f90134ef640 -1 bdev(0x55ff9e4c9800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:24.153 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:24.150+0000 7f90134ef640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-17T16:00:27.078 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T16:00:27.105 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:27.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:27.192+0000 7f44a7141640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-17T16:00:27.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:27.192+0000 7f44a7141640 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-17T16:00:27.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:27.192+0000 7f44a7141640 -1 bdev(0x55a7008a1800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:27.195 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:27.193+0000 7f44a7141640 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-17T16:00:30.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-17T16:00:30.380 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:30.470 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:30.467+0000 7fa0f0615640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-17T16:00:30.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:30.468+0000 7fa0f0615640 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-17T16:00:30.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:30.468+0000 7fa0f0615640 -1 bdev(0x55e018c77800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:30.471 INFO:teuthology.orchestra.run.smithi169.stderr:2024-09-17T16:00:30.468+0000 7fa0f0615640 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-17T16:00:33.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-17T16:00:33.666 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T16:00:33.698 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:33.698 INFO:tasks.ceph:role: osd.3 2024-09-17T16:00:33.698 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:00:33.698 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:33.774 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:33.775 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:33.775 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:33.775 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:34.274 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T16:00:34.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T16:00:34.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-17T16:00:34.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T16:00:34.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-17T16:00:34.506 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:34.506 INFO:tasks.ceph:role: osd.7 2024-09-17T16:00:34.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:00:34.506 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:34.580 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:34.581 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:34.581 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:35.042 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T16:00:35.045 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T16:00:35.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-17T16:00:35.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-17T16:00:35.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-17T16:00:35.235 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:35.235 INFO:tasks.ceph:role: osd.11 2024-09-17T16:00:35.235 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:00:35.235 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T16:00:35.308 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:35.308 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:35.309 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:35.310 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:35.310 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:35.797 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T16:00:35.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T16:00:35.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-17T16:00:35.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-17T16:00:35.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-17T16:00:36.032 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-17T16:00:36.032 INFO:tasks.ceph:role: osd.15 2024-09-17T16:00:36.032 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:00:36.032 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T16:00:36.105 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T16:00:36.105 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T16:00:36.105 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T16:00:36.105 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T16:00:36.105 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T16:00:36.106 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T16:00:36.106 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T16:00:36.106 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T16:00:36.106 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T16:00:36.106 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T16:00:36.631 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T16:00:36.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T16:00:36.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-17T16:00:36.774 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-17T16:00:36.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:36.900 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:36.897+0000 7fac41c1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T16:00:36.900 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:36.897+0000 7fac41c1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T16:00:36.900 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:36.897+0000 7fac41c1e640 -1 bdev(0x55bf39fb3800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:36.900 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:36.897+0000 7fac41c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T16:00:39.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T16:00:39.845 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:39.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:39.934+0000 7f8770bb2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-17T16:00:39.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:39.935+0000 7f8770bb2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-17T16:00:39.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:39.935+0000 7f8770bb2640 -1 bdev(0x55ae10ff1800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:39.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:39.935+0000 7f8770bb2640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-17T16:00:42.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T16:00:42.883 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:42.973 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:42.970+0000 7f8b08a3e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-17T16:00:42.973 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:42.970+0000 7f8b08a3e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-17T16:00:42.973 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:42.970+0000 7f8b08a3e640 -1 bdev(0x564d9aa67800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:42.973 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:42.971+0000 7f8b08a3e640 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-17T16:00:45.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-17T16:00:45.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:46.000 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:45.997+0000 7f99ac9bd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-17T16:00:46.000 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:45.997+0000 7f99ac9bd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-17T16:00:46.000 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:45.997+0000 7f99ac9bd640 -1 bdev(0x55fc98a83800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-17T16:00:46.000 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T16:00:45.997+0000 7f99ac9bd640 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-17T16:00:49.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-17T16:00:49.431 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T16:00:49.431 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:00:49.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-17T16:00:49.464 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:00:49.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T16:00:49.536 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:00:49.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T16:00:49.607 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:00:49.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-17T16:00:49.677 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:00:49.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-17T16:00:49.746 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:00:49.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T16:00:49.777 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:00:49.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T16:00:49.847 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:00:49.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-17T16:00:49.917 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:00:49.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-17T16:00:49.986 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:00:49.986 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T16:00:50.016 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:00:50.016 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-17T16:00:50.085 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:00:50.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-17T16:00:50.155 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:00:50.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-17T16:00:50.223 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.223 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T16:00:50.254 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T16:00:50.323 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-17T16:00:50.391 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-17T16:00:50.459 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-17T16:00:50.526 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:00:50.526 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T16:00:50.583 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T16:00:50.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T16:00:50.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T16:00:50.590 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.y] 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQByp+lmOq44BBAAD7RArTuf6XR/MJ9UsDiXQQ== 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB1p+lmcrYZGhAAVpdKVw27HpZjQjoCnScLXQ== 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB4p+lmKR+oHBAAgT0vKhGYBGY1kY2wzWVF3g== 2024-09-17T16:00:50.613 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.8] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB7p+lmR2BJLRAA2UJw7wuccxEwJYxmwHtHAg== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.12] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQB/p+lmEYZEAxAAzGdhFkgLoaepJK7Dd8w9Jg== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCDp+lmGrKEExAA/lfPYUxIGW921Gtlc2uVGg== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCGp+lmtyl3MhAAUKcBFq0ESMDpdT6M/g4Ziw== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.9] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCKp+lm9N4LJhAAsCCjpDfVLZsFJ9W9UYoKtw== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.13] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCOp+lmlNbaGRAAe1+FNxjOAKtFGrcwNsHC9A== 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2024-09-17T16:00:50.614 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCUp+lm6LffMxAA1giSEMW/DWS1iXxCBsUsjQ== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.6] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCYp+lmOcoCCRAACRXzbNku1ooHYbGXSBvaDw== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.10] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCbp+lmSJ6HCxAAcf38+6Lil0E8Tb5qQBzyIw== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.14] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCep+lm/6/wGxAAYQ3fq3BNNk8qm12gIx2zMA== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQByp+lmDMNDCRAAby8jeTXtYZnd7pY3AKX/OA== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCkp+lmtJqQNRAAXHyZgqCtEoVQPvLbodlJ0Q== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.7] 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCnp+lmRpXMNxAA7Njs1yjI3PVbOuGfeRMM0A== 2024-09-17T16:00:50.615 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.11] 2024-09-17T16:00:50.616 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCqp+lm623uORAAs2TbXcf03oksvGyGW30CcA== 2024-09-17T16:00:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.15] 2024-09-17T16:00:50.616 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCtp+lmgi2IOxAAZX1KLkVW5KoNVqYOEQAYfg== 2024-09-17T16:00:50.616 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-09-17T16:00:50.616 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQByp+lm0b79CxAAuTUE8nv0avGBQ1JlalSj9w== 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.y] 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQByp+lmOq44BBAAD7RArTuf6XR/MJ9UsDiXQQ== 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB1p+lmcrYZGhAAVpdKVw27HpZjQjoCnScLXQ== 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.4] 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB4p+lmKR+oHBAAgT0vKhGYBGY1kY2wzWVF3g== 2024-09-17T16:00:50.617 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.8] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB7p+lmR2BJLRAA2UJw7wuccxEwJYxmwHtHAg== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.12] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQB/p+lmEYZEAxAAzGdhFkgLoaepJK7Dd8w9Jg== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCDp+lmGrKEExAA/lfPYUxIGW921Gtlc2uVGg== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.5] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCGp+lmtyl3MhAAUKcBFq0ESMDpdT6M/g4Ziw== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.9] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCKp+lm9N4LJhAAsCCjpDfVLZsFJ9W9UYoKtw== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.13] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCOp+lmlNbaGRAAe1+FNxjOAKtFGrcwNsHC9A== 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-09-17T16:00:50.618 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCUp+lm6LffMxAA1giSEMW/DWS1iXxCBsUsjQ== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.6] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCYp+lmOcoCCRAACRXzbNku1ooHYbGXSBvaDw== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.10] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCbp+lmSJ6HCxAAcf38+6Lil0E8Tb5qQBzyIw== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.14] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCep+lm/6/wGxAAYQ3fq3BNNk8qm12gIx2zMA== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQByp+lmDMNDCRAAby8jeTXtYZnd7pY3AKX/OA== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.3] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCkp+lmtJqQNRAAXHyZgqCtEoVQPvLbodlJ0Q== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.7] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCnp+lmRpXMNxAA7Njs1yjI3PVbOuGfeRMM0A== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.11] 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCqp+lm623uORAAs2TbXcf03oksvGyGW30CcA== 2024-09-17T16:00:50.619 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.15] 2024-09-17T16:00:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCtp+lmgi2IOxAAZX1KLkVW5KoNVqYOEQAYfg== 2024-09-17T16:00:50.620 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-09-17T16:00:50.620 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQByp+lm0b79CxAAuTUE8nv0avGBQ1JlalSj9w== 2024-09-17T16:00:50.621 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.y] 2024-09-17T16:00:50.621 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQByp+lmOq44BBAAD7RArTuf6XR/MJ9UsDiXQQ== 2024-09-17T16:00:50.621 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQB1p+lmcrYZGhAAVpdKVw27HpZjQjoCnScLXQ== 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.4] 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQB4p+lmKR+oHBAAgT0vKhGYBGY1kY2wzWVF3g== 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.8] 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQB7p+lmR2BJLRAA2UJw7wuccxEwJYxmwHtHAg== 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.12] 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQB/p+lmEYZEAxAAzGdhFkgLoaepJK7Dd8w9Jg== 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCDp+lmGrKEExAA/lfPYUxIGW921Gtlc2uVGg== 2024-09-17T16:00:50.622 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.5] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCGp+lmtyl3MhAAUKcBFq0ESMDpdT6M/g4Ziw== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.9] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCKp+lm9N4LJhAAsCCjpDfVLZsFJ9W9UYoKtw== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.13] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCOp+lmlNbaGRAAe1+FNxjOAKtFGrcwNsHC9A== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCUp+lm6LffMxAA1giSEMW/DWS1iXxCBsUsjQ== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.6] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCYp+lmOcoCCRAACRXzbNku1ooHYbGXSBvaDw== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.10] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCbp+lmSJ6HCxAAcf38+6Lil0E8Tb5qQBzyIw== 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.14] 2024-09-17T16:00:50.623 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCep+lm/6/wGxAAYQ3fq3BNNk8qm12gIx2zMA== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQByp+lmDMNDCRAAby8jeTXtYZnd7pY3AKX/OA== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.3] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCkp+lmtJqQNRAAXHyZgqCtEoVQPvLbodlJ0Q== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.7] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCnp+lmRpXMNxAA7Njs1yjI3PVbOuGfeRMM0A== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.11] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCqp+lm623uORAAs2TbXcf03oksvGyGW30CcA== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.15] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQCtp+lmgi2IOxAAZX1KLkVW5KoNVqYOEQAYfg== 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-09-17T16:00:50.624 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQByp+lm0b79CxAAuTUE8nv0avGBQ1JlalSj9w== 2024-09-17T16:00:50.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:50.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:50.665 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:50.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.727 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.842 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.899 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:50.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.123 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.235 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.347 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:51.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:51.403 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T16:00:51.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.569 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.625 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T16:00:51.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T16:00:51.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T16:00:51.680 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T16:00:51.734 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T16:00:51.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T16:00:51.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T16:00:51.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T16:00:51.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T16:00:51.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T16:00:52.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T16:00:52.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T16:00:52.058 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T16:00:52.155 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T16:00:52.180 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:52.182 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:52.184 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T16:00:52.235 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T16:00:52.235 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T16:00:52.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T16:00:52.238 INFO:tasks.ceph.mon.a:Started 2024-09-17T16:00:52.238 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T16:00:52.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T16:00:52.240 INFO:tasks.ceph.mon.b:Started 2024-09-17T16:00:52.241 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T16:00:52.241 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T16:00:52.277 INFO:tasks.ceph.mon.c:Started 2024-09-17T16:00:52.277 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T16:00:52.278 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-17T16:00:52.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-17T16:00:52.280 INFO:tasks.ceph.mgr.y:Started 2024-09-17T16:00:52.280 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T16:00:52.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T16:00:52.283 INFO:tasks.ceph.mgr.x:Started 2024-09-17T16:00:52.283 DEBUG:tasks.ceph:set 0 configs 2024-09-17T16:00:52.283 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T16:00:52.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T16:01:02.692 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-09-17T16:01:02.705 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T16:01:02.705 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T16:01:02.705 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:01:02.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T16:01:02.732 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:01:02.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T16:01:02.799 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:01:02.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-17T16:01:02.866 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-09-17T16:01:02.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-17T16:01:02.932 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:01:02.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T16:01:02.966 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:01:02.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T16:01:03.036 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:01:03.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-17T16:01:03.105 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-17T16:01:03.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-17T16:01:03.173 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:01:03.173 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T16:01:03.206 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:01:03.206 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-17T16:01:03.276 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:01:03.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-17T16:01:03.345 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-09-17T16:01:03.345 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-17T16:01:03.413 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:01:03.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T16:01:03.439 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:01:03.440 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-17T16:01:03.505 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:01:03.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-17T16:01:03.569 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T16:01:03.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-17T16:01:03.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 282c4269-7e53-43fc-af4b-3b133d82837c 0 2024-09-17T16:01:03.859 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-09-17T16:01:03.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 66e20d10-2bb0-4130-93a5-45bcdd2de605 1 2024-09-17T16:01:04.058 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-17T16:01:04.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new c6ec9e20-133a-471b-b63c-6453a7bde86c 2 2024-09-17T16:01:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-09-17T16:01:04.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 3491a731-ca96-4981-be73-e005e00c315f 3 2024-09-17T16:01:04.459 INFO:teuthology.orchestra.run.smithi181.stdout:3 2024-09-17T16:01:04.471 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 3bdfaa34-bc10-4cec-b5c2-86e05d444ffd 4 2024-09-17T16:01:04.656 INFO:teuthology.orchestra.run.smithi181.stdout:4 2024-09-17T16:01:04.667 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2 5 2024-09-17T16:01:04.849 INFO:teuthology.orchestra.run.smithi181.stdout:5 2024-09-17T16:01:04.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new ddd3edea-260a-4ec9-8533-841ed933fdb8 6 2024-09-17T16:01:05.041 INFO:teuthology.orchestra.run.smithi181.stdout:6 2024-09-17T16:01:05.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 7afc1a18-98c2-4a58-a09d-1b4f474d4dc4 7 2024-09-17T16:01:05.234 INFO:teuthology.orchestra.run.smithi181.stdout:7 2024-09-17T16:01:05.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 3e45f80f-856b-4d3f-b002-255f16a14f6b 8 2024-09-17T16:01:05.427 INFO:teuthology.orchestra.run.smithi181.stdout:8 2024-09-17T16:01:05.438 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new b415c423-24de-4d86-bec4-ea330364ff96 9 2024-09-17T16:01:05.620 INFO:teuthology.orchestra.run.smithi181.stdout:9 2024-09-17T16:01:05.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new e471e1cb-9c0b-4c99-8622-59eb7956770d 10 2024-09-17T16:01:05.814 INFO:teuthology.orchestra.run.smithi181.stdout:10 2024-09-17T16:01:05.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new c2bf4255-7ce1-4673-bedf-fd055d841fac 11 2024-09-17T16:01:06.022 INFO:teuthology.orchestra.run.smithi181.stdout:11 2024-09-17T16:01:06.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 4c16f3ee-f427-41d5-b0ba-9e676ddbe032 12 2024-09-17T16:01:06.216 INFO:teuthology.orchestra.run.smithi181.stdout:12 2024-09-17T16:01:06.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 0a4386aa-1a59-44c8-8d1e-7d43bf10de2e 13 2024-09-17T16:01:06.410 INFO:teuthology.orchestra.run.smithi181.stdout:13 2024-09-17T16:01:06.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 4c54e35c-e226-4d8c-bad0-51dd1cd95fe0 14 2024-09-17T16:01:06.603 INFO:teuthology.orchestra.run.smithi181.stdout:14 2024-09-17T16:01:06.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 0281e750-f74a-4760-bbef-fbfa4127d06b 15 2024-09-17T16:01:06.797 INFO:teuthology.orchestra.run.smithi181.stdout:15 2024-09-17T16:01:06.808 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T16:01:06.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T16:01:06.810 INFO:tasks.ceph.osd.0:Started 2024-09-17T16:01:06.810 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T16:01:06.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T16:01:06.812 INFO:tasks.ceph.osd.4:Started 2024-09-17T16:01:06.812 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-17T16:01:06.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-17T16:01:06.814 INFO:tasks.ceph.osd.8:Started 2024-09-17T16:01:06.814 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-17T16:01:06.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-17T16:01:06.817 INFO:tasks.ceph.osd.12:Started 2024-09-17T16:01:06.817 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T16:01:06.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T16:01:06.820 INFO:tasks.ceph.osd.1:Started 2024-09-17T16:01:06.820 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T16:01:06.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T16:01:06.822 INFO:tasks.ceph.osd.5:Started 2024-09-17T16:01:06.822 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-17T16:01:06.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-17T16:01:06.824 INFO:tasks.ceph.osd.9:Started 2024-09-17T16:01:06.825 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-17T16:01:06.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-17T16:01:06.829 INFO:tasks.ceph.osd.13:Started 2024-09-17T16:01:06.830 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T16:01:06.830 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T16:01:06.832 INFO:tasks.ceph.osd.2:Started 2024-09-17T16:01:06.832 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-17T16:01:06.832 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-17T16:01:06.834 INFO:tasks.ceph.osd.6:Started 2024-09-17T16:01:06.834 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-17T16:01:06.835 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-17T16:01:06.837 INFO:tasks.ceph.osd.10:Started 2024-09-17T16:01:06.837 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-17T16:01:06.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-17T16:01:06.839 INFO:tasks.ceph.osd.14:Started 2024-09-17T16:01:06.839 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T16:01:06.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T16:01:06.841 INFO:tasks.ceph.osd.3:Started 2024-09-17T16:01:06.842 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-17T16:01:06.842 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-17T16:01:06.843 INFO:tasks.ceph.osd.7:Started 2024-09-17T16:01:06.844 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-17T16:01:06.844 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-17T16:01:06.846 INFO:tasks.ceph.osd.11:Started 2024-09-17T16:01:06.846 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-17T16:01:06.846 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-17T16:01:06.848 INFO:tasks.ceph.osd.15:Started 2024-09-17T16:01:06.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:01:07.051 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:07.051 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":18,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:01:06.794253+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:01:07.062 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T16:01:07.062 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T16:01:07.805 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:07.803+0000 7f7cdf422640 -1 Falling back to public interface 2024-09-17T16:01:07.948 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-17T16:01:07.946+0000 7fc74ee0e640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T16:01:07.950 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-17T16:01:07.948+0000 7fc74ee0e640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T16:01:08.044 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:08.042+0000 7fa4d3e0d640 -1 Falling back to public interface 2024-09-17T16:01:08.044 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:08.042+0000 7f891c5a2640 -1 Falling back to public interface 2024-09-17T16:01:08.044 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:08.042+0000 7fa4d2812640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-09-17T16:01:08.045 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:08.042+0000 7f891b54a640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-09-17T16:01:08.049 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:08.047+0000 7f81ff831640 -1 Falling back to public interface 2024-09-17T16:01:08.052 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:08.049+0000 7f12eb818640 -1 Falling back to public interface 2024-09-17T16:01:08.055 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:08.053+0000 7f619b80d640 -1 Falling back to public interface 2024-09-17T16:01:08.055 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:08.052+0000 7f8d2d40b640 -1 Falling back to public interface 2024-09-17T16:01:08.270 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:08.267+0000 7f440b420640 -1 Falling back to public interface 2024-09-17T16:01:08.271 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:08.268+0000 7f45ef00b640 -1 Falling back to public interface 2024-09-17T16:01:08.305 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:08.303+0000 7f84aaed8640 -1 Falling back to public interface 2024-09-17T16:01:08.305 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:08.302+0000 7f91c0042640 -1 Falling back to public interface 2024-09-17T16:01:08.310 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:08.308+0000 7f9a05e1b640 -1 Falling back to public interface 2024-09-17T16:01:08.314 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:08.313+0000 7fd1ada13640 -1 Falling back to public interface 2024-09-17T16:01:08.522 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:08.519+0000 7fd1de60c640 -1 Falling back to public interface 2024-09-17T16:01:08.533 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:08.531+0000 7f766a91b640 -1 Falling back to public interface 2024-09-17T16:01:08.539 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:08.536+0000 7f5d6f015640 -1 Falling back to public interface 2024-09-17T16:01:08.671 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T16:01:08.984 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-17T16:01:08.984 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":18,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:01:06.794253+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:01:08.995 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-17T16:01:11.652 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:11.650+0000 7fa4d3e0d640 -1 osd.1 0 log_to_monitors true 2024-09-17T16:01:11.781 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:11.779+0000 7f440b420640 -1 osd.5 0 log_to_monitors true 2024-09-17T16:01:11.855 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:11.853+0000 7f12eb818640 -1 osd.9 0 log_to_monitors true 2024-09-17T16:01:11.859 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:11.857+0000 7f619b80d640 -1 osd.2 0 log_to_monitors true 2024-09-17T16:01:11.870 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:11.868+0000 7f7cdf422640 -1 osd.14 0 log_to_monitors true 2024-09-17T16:01:11.905 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:11.903+0000 7f891c5a2640 -1 osd.13 0 log_to_monitors true 2024-09-17T16:01:12.027 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:12.025+0000 7f45ef00b640 -1 osd.0 0 log_to_monitors true 2024-09-17T16:01:12.064 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:12.063+0000 7f84aaed8640 -1 osd.15 0 log_to_monitors true 2024-09-17T16:01:12.087 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:12.084+0000 7f8d2d40b640 -1 osd.4 0 log_to_monitors true 2024-09-17T16:01:12.095 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:12.093+0000 7f81ff831640 -1 osd.10 0 log_to_monitors true 2024-09-17T16:01:12.368 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:12.366+0000 7f9a05e1b640 -1 osd.3 0 log_to_monitors true 2024-09-17T16:01:12.369 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:12.366+0000 7f5d6f015640 -1 osd.8 0 log_to_monitors true 2024-09-17T16:01:12.531 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:12.530+0000 7f766a91b640 -1 osd.11 0 log_to_monitors true 2024-09-17T16:01:12.539 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:12.537+0000 7fd1de60c640 -1 osd.6 0 log_to_monitors true 2024-09-17T16:01:12.621 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:12.619+0000 7fd1ada13640 -1 osd.7 0 log_to_monitors true 2024-09-17T16:01:12.843 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:12.840+0000 7f91c0042640 -1 osd.12 0 log_to_monitors true 2024-09-17T16:01:13.301 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:13.299+0000 7fa4cf7f9640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T16:01:13.317 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:13.315+0000 7fa4c7618640 -1 osd.1 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.730 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:13.728+0000 7f4406d11640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T16:01:13.733 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:13.732+0000 7f9a01808640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T16:01:13.735 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:13.733+0000 7f12e7224640 -1 osd.9 0 waiting for initial osdmap 2024-09-17T16:01:13.736 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:13.734+0000 7f4402b38640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.738 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:13.736+0000 7f7cdae0f640 -1 osd.14 0 waiting for initial osdmap 2024-09-17T16:01:13.738 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:13.737+0000 7f76668aa640 -1 osd.11 0 waiting for initial osdmap 2024-09-17T16:01:13.739 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:13.737+0000 7f84a6e67640 -1 osd.15 0 waiting for initial osdmap 2024-09-17T16:01:13.740 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:13.738+0000 7f12e304b640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.741 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:13.738+0000 7f5d6a8f5640 -1 osd.8 0 waiting for initial osdmap 2024-09-17T16:01:13.741 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:13.739+0000 7f99fd62f640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.747 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:13.745+0000 7f619721c640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T16:01:13.747 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:13.745+0000 7fd1d9fd9640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T16:01:13.750 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:13.749+0000 7fd1a9326640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T16:01:13.752 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:13.749+0000 7fd1d5e00640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.752 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:13.749+0000 7f5d6671c640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.753 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:13.750+0000 7f6193043640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.756 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:13.754+0000 7fd1a514d640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.756 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:13.754+0000 7f7cd6c36640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.757 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:13.755+0000 7f76626d1640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:13.757 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:13.755+0000 7f84a2c8e640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:14.290 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:14.288+0000 7f8918531640 -1 osd.13 0 waiting for initial osdmap 2024-09-17T16:01:14.304 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:14.302+0000 7f8910350640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:14.330 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:14.328+0000 7f81fad85640 -1 osd.10 0 waiting for initial osdmap 2024-09-17T16:01:14.341 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:14.339+0000 7f81f2ba4640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:14.457 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:14.454+0000 7f45eb230640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T16:01:14.469 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:14.466+0000 7f45e283b640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:14.479 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:14.476+0000 7f8d295a6640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T16:01:14.490 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:14.487+0000 7f8d20bb1640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:15.296 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:15.293+0000 7f91bbfd1640 -1 osd.12 0 waiting for initial osdmap 2024-09-17T16:01:15.307 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:15.304+0000 7f91b3df0640 -1 osd.12 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:01:16.603 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T16:01:16.915 INFO:teuthology.misc.health.smithi006.stdout: 2024-09-17T16:01:16.916 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":24,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:01:16.732182+0000","last_up_change":"2024-09-17T16:01:15.727345+0000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6819","nonce":3415295348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6821","nonce":3415295348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6825","nonce":3415295348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6823","nonce":3415295348}]},"public_addr":"172.21.15.181:6819/3415295348","cluster_addr":"172.21.15.181:6821/3415295348","heartbeat_back_addr":"172.21.15.181:6825/3415295348","heartbeat_front_addr":"172.21.15.181:6823/3415295348","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.014883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.677746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.838957+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.321698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.045654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.762032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.517939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.664321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.417977+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.810458+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.124128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.554475+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.868839+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.912548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.882601+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.090747+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:01:16.927 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-17T16:01:16.928 INFO:tasks.ceph:Creating RBD pool 2024-09-17T16:01:16.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-17T16:01:17.746 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2024-09-17T16:01:17.762 DEBUG:teuthology.orchestra.run.smithi006:> rbd --cluster ceph pool init rbd 2024-09-17T16:01:20.776 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T16:01:20.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T16:01:20.777 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T16:01:21.096 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:21.109 INFO:teuthology.orchestra.run.smithi006.stdout:[{"version":7,"timestamp":"2024-09-17T16:01:15.295610+0000","name":"","changes":[{"name":"osd.12/osd_mclock_max_capacity_iops_ssd","new_value":"9033.957925"}]}] 2024-09-17T16:01:21.109 INFO:tasks.ceph_manager:config epoch is 7 2024-09-17T16:01:21.109 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T16:01:21.110 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T16:01:21.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T16:01:21.466 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:21.476 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1879267560},{"type":"v1","addr":"172.21.15.181:6801","nonce":1879267560}]},"active_addr":"172.21.15.181:6801/1879267560","active_change":"2024-09-17T16:01:06.929236+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4101,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3363531762}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":372584441}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":1292673518}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":1276365758}]}]} 2024-09-17T16:01:21.482 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T16:01:21.482 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T16:01:21.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:01:21.787 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:21.788 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:01:20.758168+0000","last_up_change":"2024-09-17T16:01:15.727345+0000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6819","nonce":3415295348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6821","nonce":3415295348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6825","nonce":3415295348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6823","nonce":3415295348}]},"public_addr":"172.21.15.181:6819/3415295348","cluster_addr":"172.21.15.181:6821/3415295348","heartbeat_back_addr":"172.21.15.181:6825/3415295348","heartbeat_front_addr":"172.21.15.181:6823/3415295348","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.014883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.677746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.838957+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.321698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.045654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.762032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.517939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.664321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.417977+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.810458+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.124128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.554475+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.868839+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.912548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.882601+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.090747+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:01:21.800 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T16:01:21.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:01:22.105 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:22.105 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":28,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:01:20.758168+0000","last_up_change":"2024-09-17T16:01:15.727345+0000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.0399999618530273,"score_stable":6.0399999618530273,"optimal_score":0.56000000238418579,"raw_score_acting":3.380000114440918,"raw_score_stable":3.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6819","nonce":3415295348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6821","nonce":3415295348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6825","nonce":3415295348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6823","nonce":3415295348}]},"public_addr":"172.21.15.181:6819/3415295348","cluster_addr":"172.21.15.181:6821/3415295348","heartbeat_back_addr":"172.21.15.181:6825/3415295348","heartbeat_front_addr":"172.21.15.181:6823/3415295348","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.014883+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.677746+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.838957+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.321698+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.045654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.762032+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.517939+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.664321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.417977+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.810458+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.124128+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.554475+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.868839+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.912548+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:12.882601+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.090747+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:01:22.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T16:01:22.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T16:01:22.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T16:01:22.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T16:01:22.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T16:01:22.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T16:01:22.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-17T16:01:22.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-17T16:01:22.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-17T16:01:22.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-17T16:01:22.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-17T16:01:22.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-17T16:01:22.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-17T16:01:22.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-17T16:01:22.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-17T16:01:22.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-17T16:01:22.544 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T16:01:22.563 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-17T16:01:22.568 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T16:01:22.595 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-17T16:01:22.609 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-17T16:01:22.618 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T16:01:22.619 INFO:teuthology.orchestra.run.smithi006.stdout:90194313220 2024-09-17T16:01:22.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T16:01:22.620 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T16:01:22.625 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:22.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-17T16:01:22.633 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T16:01:22.634 INFO:teuthology.orchestra.run.smithi006.stdout:98784247813 2024-09-17T16:01:22.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-17T16:01:22.636 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:22.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-17T16:01:22.640 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-17T16:01:22.655 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-17T16:01:22.768 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:22.766+0000 7f926dd4f640 0 --2- 172.21.15.6:0/3187033209 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9260050080 0x7f9260051260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:01:22.793 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T16:01:22.908 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:22.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T16:01:23.283 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:23.281+0000 7f345f7fe640 0 --2- 172.21.15.6:0/428862381 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f346015ee80 0x7f34601fffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T16:01:23.434 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.469 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.7 2024-09-17T16:01:23.600 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.15 2024-09-17T16:01:23.755 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.777 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.4 2024-09-17T16:01:23.780 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.790 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-09-17T16:01:23.814 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.814 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.817 INFO:teuthology.orchestra.run.smithi006.stdout:90194313220 2024-09-17T16:01:23.820 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.5 2024-09-17T16:01:23.844 INFO:teuthology.orchestra.run.smithi006.stdout:94489280515 2024-09-17T16:01:23.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.1 2024-09-17T16:01:23.850 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.854 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.857 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-17T16:01:23.858 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.11 2024-09-17T16:01:23.861 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.867 INFO:teuthology.orchestra.run.smithi006.stdout:98784247813 2024-09-17T16:01:23.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-17T16:01:23.874 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.875 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.13 2024-09-17T16:01:23.881 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.886 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.12 2024-09-17T16:01:23.886 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-17T16:01:23.897 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.923 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.930 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-09-17T16:01:23.937 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.944 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2024-09-17T16:01:23.944 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.955 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.961 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:23.969 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.2 2024-09-17T16:01:23.969 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:23.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.14 2024-09-17T16:01:23.975 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:24.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T16:01:24.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-17T16:01:24.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T16:01:24.806 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:24.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.7 2024-09-17T16:01:24.821 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:24.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T16:01:24.956 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:24.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.15 2024-09-17T16:01:24.970 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:25.114 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:25.128 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-09-17T16:01:25.128 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:25.189 INFO:teuthology.orchestra.run.smithi006.stdout:94489280516 2024-09-17T16:01:25.202 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2024-09-17T16:01:25.202 DEBUG:teuthology.parallel:result is None 2024-09-17T16:01:25.202 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T16:01:25.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:25.505 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:25.517 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T16:01:24.937660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445192,"kb_used_data":4320,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504248,"statfs":{"total":1546188226560,"available":1545732349952,"internally_reserved":0,"allocated":4423680,"data_stored":2970150,"data_compressed":295651,"data_compressed_allocated":458752,"data_compressed_original":1794048,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.177844"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767115+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:20.767115+0000","last_peered":"2024-09-17T16:01:20.767115+0000","last_clean":"2024-09-17T16:01:20.767115+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:20.767115+0000","last_undegraded":"2024-09-17T16:01:20.767115+0000","last_fullsized":"2024-09-17T16:01:20.767115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.769493+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:20.769493+0000","last_peered":"2024-09-17T16:01:20.769493+0000","last_clean":"2024-09-17T16:01:20.769493+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:20.769493+0000","last_undegraded":"2024-09-17T16:01:20.769493+0000","last_fullsized":"2024-09-17T16:01:20.769493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.762194+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:20.762194+0000","last_peered":"2024-09-17T16:01:20.762194+0000","last_clean":"2024-09-17T16:01:20.762194+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:20.762194+0000","last_undegraded":"2024-09-17T16:01:20.762194+0000","last_fullsized":"2024-09-17T16:01:20.762194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.761941+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:20.761941+0000","last_peered":"2024-09-17T16:01:20.761941+0000","last_clean":"2024-09-17T16:01:20.761941+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:20.761941+0000","last_undegraded":"2024-09-17T16:01:20.761941+0000","last_fullsized":"2024-09-17T16:01:20.761941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767827+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:20.767827+0000","last_peered":"2024-09-17T16:01:20.767827+0000","last_clean":"2024-09-17T16:01:20.767827+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:20.767827+0000","last_undegraded":"2024-09-17T16:01:20.767827+0000","last_fullsized":"2024-09-17T16:01:20.767827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.267648+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:21.267648+0000","last_peered":"2024-09-17T16:01:21.267648+0000","last_clean":"2024-09-17T16:01:21.267648+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:21.267648+0000","last_undegraded":"2024-09-17T16:01:21.267648+0000","last_fullsized":"2024-09-17T16:01:21.267648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767084+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:20.767084+0000","last_peered":"2024-09-17T16:01:20.767084+0000","last_clean":"2024-09-17T16:01:20.767084+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:20.767084+0000","last_undegraded":"2024-09-17T16:01:20.767084+0000","last_fullsized":"2024-09-17T16:01:20.767084+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.331234+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:21.331234+0000","last_peered":"2024-09-17T16:01:21.331234+0000","last_clean":"2024-09-17T16:01:21.331234+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:21.331234+0000","last_undegraded":"2024-09-17T16:01:21.331234+0000","last_fullsized":"2024-09-17T16:01:21.331234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.330837+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:21.330837+0000","last_peered":"2024-09-17T16:01:21.330837+0000","last_clean":"2024-09-17T16:01:21.330837+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:21.330837+0000","last_undegraded":"2024-09-17T16:01:21.330837+0000","last_fullsized":"2024-09-17T16:01:21.330837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":101696,"data_compressed":11658,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:25.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:25.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:25.821 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:25.833 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T16:01:24.937660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445192,"kb_used_data":4320,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504248,"statfs":{"total":1546188226560,"available":1545732349952,"internally_reserved":0,"allocated":4423680,"data_stored":2970150,"data_compressed":295651,"data_compressed_allocated":458752,"data_compressed_original":1794048,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.177844"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767115+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:20.767115+0000","last_peered":"2024-09-17T16:01:20.767115+0000","last_clean":"2024-09-17T16:01:20.767115+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:20.767115+0000","last_undegraded":"2024-09-17T16:01:20.767115+0000","last_fullsized":"2024-09-17T16:01:20.767115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.769493+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:20.769493+0000","last_peered":"2024-09-17T16:01:20.769493+0000","last_clean":"2024-09-17T16:01:20.769493+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:20.769493+0000","last_undegraded":"2024-09-17T16:01:20.769493+0000","last_fullsized":"2024-09-17T16:01:20.769493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.762194+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:20.762194+0000","last_peered":"2024-09-17T16:01:20.762194+0000","last_clean":"2024-09-17T16:01:20.762194+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:20.762194+0000","last_undegraded":"2024-09-17T16:01:20.762194+0000","last_fullsized":"2024-09-17T16:01:20.762194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.761941+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:20.761941+0000","last_peered":"2024-09-17T16:01:20.761941+0000","last_clean":"2024-09-17T16:01:20.761941+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:20.761941+0000","last_undegraded":"2024-09-17T16:01:20.761941+0000","last_fullsized":"2024-09-17T16:01:20.761941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767827+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:20.767827+0000","last_peered":"2024-09-17T16:01:20.767827+0000","last_clean":"2024-09-17T16:01:20.767827+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:20.767827+0000","last_undegraded":"2024-09-17T16:01:20.767827+0000","last_fullsized":"2024-09-17T16:01:20.767827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.267648+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:21.267648+0000","last_peered":"2024-09-17T16:01:21.267648+0000","last_clean":"2024-09-17T16:01:21.267648+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:21.267648+0000","last_undegraded":"2024-09-17T16:01:21.267648+0000","last_fullsized":"2024-09-17T16:01:21.267648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767084+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:20.767084+0000","last_peered":"2024-09-17T16:01:20.767084+0000","last_clean":"2024-09-17T16:01:20.767084+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:20.767084+0000","last_undegraded":"2024-09-17T16:01:20.767084+0000","last_fullsized":"2024-09-17T16:01:20.767084+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.331234+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:21.331234+0000","last_peered":"2024-09-17T16:01:21.331234+0000","last_clean":"2024-09-17T16:01:21.331234+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:21.331234+0000","last_undegraded":"2024-09-17T16:01:21.331234+0000","last_fullsized":"2024-09-17T16:01:21.331234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.330837+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:21.330837+0000","last_peered":"2024-09-17T16:01:21.330837+0000","last_clean":"2024-09-17T16:01:21.330837+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:21.330837+0000","last_undegraded":"2024-09-17T16:01:21.330837+0000","last_fullsized":"2024-09-17T16:01:21.330837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":101696,"data_compressed":11658,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:25.834 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T16:01:25.834 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T16:01:25.834 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T16:01:25.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T16:01:26.181 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:26.181 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T16:01:26.194 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T16:01:26.195 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-17T16:01:26.205 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-17T16:01:26.205 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T16:01:26.205 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-17T16:01:26.206 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-17T16:01:26.206 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T16:01:26.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:26.507 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:26.507 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:26.519 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T16:01:24.937660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445192,"kb_used_data":4320,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504248,"statfs":{"total":1546188226560,"available":1545732349952,"internally_reserved":0,"allocated":4423680,"data_stored":2970150,"data_compressed":295651,"data_compressed_allocated":458752,"data_compressed_original":1794048,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.177844"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767115+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:20.767115+0000","last_peered":"2024-09-17T16:01:20.767115+0000","last_clean":"2024-09-17T16:01:20.767115+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:20.767115+0000","last_undegraded":"2024-09-17T16:01:20.767115+0000","last_fullsized":"2024-09-17T16:01:20.767115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.769493+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:20.769493+0000","last_peered":"2024-09-17T16:01:20.769493+0000","last_clean":"2024-09-17T16:01:20.769493+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:20.769493+0000","last_undegraded":"2024-09-17T16:01:20.769493+0000","last_fullsized":"2024-09-17T16:01:20.769493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.762194+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:20.762194+0000","last_peered":"2024-09-17T16:01:20.762194+0000","last_clean":"2024-09-17T16:01:20.762194+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:20.762194+0000","last_undegraded":"2024-09-17T16:01:20.762194+0000","last_fullsized":"2024-09-17T16:01:20.762194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.761941+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:20.761941+0000","last_peered":"2024-09-17T16:01:20.761941+0000","last_clean":"2024-09-17T16:01:20.761941+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:20.761941+0000","last_undegraded":"2024-09-17T16:01:20.761941+0000","last_fullsized":"2024-09-17T16:01:20.761941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767827+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:20.767827+0000","last_peered":"2024-09-17T16:01:20.767827+0000","last_clean":"2024-09-17T16:01:20.767827+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:20.767827+0000","last_undegraded":"2024-09-17T16:01:20.767827+0000","last_fullsized":"2024-09-17T16:01:20.767827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.267648+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:21.267648+0000","last_peered":"2024-09-17T16:01:21.267648+0000","last_clean":"2024-09-17T16:01:21.267648+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:21.267648+0000","last_undegraded":"2024-09-17T16:01:21.267648+0000","last_fullsized":"2024-09-17T16:01:21.267648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767084+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:20.767084+0000","last_peered":"2024-09-17T16:01:20.767084+0000","last_clean":"2024-09-17T16:01:20.767084+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:20.767084+0000","last_undegraded":"2024-09-17T16:01:20.767084+0000","last_fullsized":"2024-09-17T16:01:20.767084+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.331234+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:21.331234+0000","last_peered":"2024-09-17T16:01:21.331234+0000","last_clean":"2024-09-17T16:01:21.331234+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:21.331234+0000","last_undegraded":"2024-09-17T16:01:21.331234+0000","last_fullsized":"2024-09-17T16:01:21.331234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.330837+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:21.330837+0000","last_peered":"2024-09-17T16:01:21.330837+0000","last_clean":"2024-09-17T16:01:21.330837+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:21.330837+0000","last_undegraded":"2024-09-17T16:01:21.330837+0000","last_fullsized":"2024-09-17T16:01:21.330837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":101696,"data_compressed":11658,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:26.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:26.822 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:26.834 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-17T16:01:24.937660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":445192,"kb_used_data":4320,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504248,"statfs":{"total":1546188226560,"available":1545732349952,"internally_reserved":0,"allocated":4423680,"data_stored":2970150,"data_compressed":295651,"data_compressed_allocated":458752,"data_compressed_original":1794048,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.177844"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767115+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:20.767115+0000","last_peered":"2024-09-17T16:01:20.767115+0000","last_clean":"2024-09-17T16:01:20.767115+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:20.767115+0000","last_undegraded":"2024-09-17T16:01:20.767115+0000","last_fullsized":"2024-09-17T16:01:20.767115+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.769493+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:20.769493+0000","last_peered":"2024-09-17T16:01:20.769493+0000","last_clean":"2024-09-17T16:01:20.769493+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:20.769493+0000","last_undegraded":"2024-09-17T16:01:20.769493+0000","last_fullsized":"2024-09-17T16:01:20.769493+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.762194+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:20.762194+0000","last_peered":"2024-09-17T16:01:20.762194+0000","last_clean":"2024-09-17T16:01:20.762194+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:20.762194+0000","last_undegraded":"2024-09-17T16:01:20.762194+0000","last_fullsized":"2024-09-17T16:01:20.762194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.761941+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:20.761941+0000","last_peered":"2024-09-17T16:01:20.761941+0000","last_clean":"2024-09-17T16:01:20.761941+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:20.761941+0000","last_undegraded":"2024-09-17T16:01:20.761941+0000","last_fullsized":"2024-09-17T16:01:20.761941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767827+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:20.767827+0000","last_peered":"2024-09-17T16:01:20.767827+0000","last_clean":"2024-09-17T16:01:20.767827+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:20.767827+0000","last_undegraded":"2024-09-17T16:01:20.767827+0000","last_fullsized":"2024-09-17T16:01:20.767827+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.267648+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:21.267648+0000","last_peered":"2024-09-17T16:01:21.267648+0000","last_clean":"2024-09-17T16:01:21.267648+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:21.267648+0000","last_undegraded":"2024-09-17T16:01:21.267648+0000","last_fullsized":"2024-09-17T16:01:21.267648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:20.767084+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:20.767084+0000","last_peered":"2024-09-17T16:01:20.767084+0000","last_clean":"2024-09-17T16:01:20.767084+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:20.767084+0000","last_undegraded":"2024-09-17T16:01:20.767084+0000","last_fullsized":"2024-09-17T16:01:20.767084+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.331234+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:21.331234+0000","last_peered":"2024-09-17T16:01:21.331234+0000","last_clean":"2024-09-17T16:01:21.331234+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:21.331234+0000","last_undegraded":"2024-09-17T16:01:21.331234+0000","last_fullsized":"2024-09-17T16:01:21.331234+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":155,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-17T16:01:21.330837+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:21.330837+0000","last_peered":"2024-09-17T16:01:21.330837+0000","last_clean":"2024-09-17T16:01:21.330837+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:21.330837+0000","last_undegraded":"2024-09-17T16:01:21.330837+0000","last_fullsized":"2024-09-17T16:01:21.330837+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27272,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":245760,"data_stored":101696,"data_compressed":11658,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,7,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,8,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6,7,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120723,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":120704,"data_compressed":17533,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":327680,"data_stored":649632,"data_compressed":28032,"data_compressed_allocated":61440,"data_compressed_original":561152,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:26.835 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T16:01:26.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-17T16:01:27.142 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 28 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:fsid 4345ca04-2628-4bef-a537-6837268b96c9 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-09-17T16:01:02.569356+0000 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-09-17T16:01:20.758168+0000 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:crush_version 6 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:full_ratio 0.95 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:backfillfull_ratio 0.9 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:nearfull_ratio 0.85 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:require_min_compat_client luminous 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:min_compat_client jewel 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:require_osd_release squid 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:stretch_mode_enabled false 2024-09-17T16:01:27.143 INFO:teuthology.orchestra.run.smithi006.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 6.04 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout: removed_snaps_queue [2~1] 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:max_osd 16 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6808/865094279,v1:172.21.15.6:6809/865094279] [v2:172.21.15.6:6810/865094279,v1:172.21.15.6:6811/865094279] exists,up 282c4269-7e53-43fc-af4b-3b133d82837c 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6801/1460756480,v1:172.21.15.50:6803/1460756480] [v2:172.21.15.50:6805/1460756480,v1:172.21.15.50:6807/1460756480] exists,up 66e20d10-2bb0-4130-93a5-45bcdd2de605 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6816/1905413450,v1:172.21.15.169:6817/1905413450] [v2:172.21.15.169:6818/1905413450,v1:172.21.15.169:6819/1905413450] exists,up c6ec9e20-133a-471b-b63c-6453a7bde86c 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6810/2918557179,v1:172.21.15.181:6811/2918557179] [v2:172.21.15.181:6812/2918557179,v1:172.21.15.181:6813/2918557179] exists,up 3491a731-ca96-4981-be73-e005e00c315f 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6800/2257685015,v1:172.21.15.6:6801/2257685015] [v2:172.21.15.6:6802/2257685015,v1:172.21.15.6:6803/2257685015] exists,up 3bdfaa34-bc10-4cec-b5c2-86e05d444ffd 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6824/2399951648,v1:172.21.15.50:6825/2399951648] [v2:172.21.15.50:6826/2399951648,v1:172.21.15.50:6827/2399951648] exists,up 5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6824/1225429423,v1:172.21.15.169:6825/1225429423] [v2:172.21.15.169:6826/1225429423,v1:172.21.15.169:6827/1225429423] exists,up ddd3edea-260a-4ec9-8533-841ed933fdb8 2024-09-17T16:01:27.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6818/3415295348,v1:172.21.15.181:6819/3415295348] [v2:172.21.15.181:6820/3415295348,v1:172.21.15.181:6821/3415295348] exists,up 7afc1a18-98c2-4a58-a09d-1b4f474d4dc4 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6824/2216439459,v1:172.21.15.6:6825/2216439459] [v2:172.21.15.6:6826/2216439459,v1:172.21.15.6:6827/2216439459] exists,up 3e45f80f-856b-4d3f-b002-255f16a14f6b 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/1727176716,v1:172.21.15.50:6817/1727176716] [v2:172.21.15.50:6818/1727176716,v1:172.21.15.50:6819/1727176716] exists,up b415c423-24de-4d86-bec4-ea330364ff96 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6808/3604183526,v1:172.21.15.169:6809/3604183526] [v2:172.21.15.169:6810/3604183526,v1:172.21.15.169:6811/3604183526] exists,up e471e1cb-9c0b-4c99-8622-59eb7956770d 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6826/3420013159,v1:172.21.15.181:6827/3420013159] [v2:172.21.15.181:6828/3420013159,v1:172.21.15.181:6829/3420013159] exists,up c2bf4255-7ce1-4673-bedf-fd055d841fac 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6816/2310487797,v1:172.21.15.6:6817/2310487797] [v2:172.21.15.6:6818/2310487797,v1:172.21.15.6:6819/2310487797] exists,up 4c16f3ee-f427-41d5-b0ba-9e676ddbe032 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/1210437211,v1:172.21.15.50:6802/1210437211] [v2:172.21.15.50:6804/1210437211,v1:172.21.15.50:6806/1210437211] exists,up 0a4386aa-1a59-44c8-8d1e-7d43bf10de2e 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] [v2:172.21.15.169:6802/4170280864,v1:172.21.15.169:6803/4170280864] exists,up 4c54e35c-e226-4d8c-bad0-51dd1cd95fe0 2024-09-17T16:01:27.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] [v2:172.21.15.181:6804/2815528414,v1:172.21.15.181:6805/2815528414] exists,up 0281e750-f74a-4760-bbef-fbfa4127d06b 2024-09-17T16:01:27.155 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6808/865094279,v1:172.21.15.6:6809/865094279] [v2:172.21.15.6:6810/865094279,v1:172.21.15.6:6811/865094279] exists,up 282c4269-7e53-43fc-af4b-3b133d82837c', 'osd.1 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6801/1460756480,v1:172.21.15.50:6803/1460756480] [v2:172.21.15.50:6805/1460756480,v1:172.21.15.50:6807/1460756480] exists,up 66e20d10-2bb0-4130-93a5-45bcdd2de605', 'osd.2 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6816/1905413450,v1:172.21.15.169:6817/1905413450] [v2:172.21.15.169:6818/1905413450,v1:172.21.15.169:6819/1905413450] exists,up c6ec9e20-133a-471b-b63c-6453a7bde86c', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6810/2918557179,v1:172.21.15.181:6811/2918557179] [v2:172.21.15.181:6812/2918557179,v1:172.21.15.181:6813/2918557179] exists,up 3491a731-ca96-4981-be73-e005e00c315f', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6800/2257685015,v1:172.21.15.6:6801/2257685015] [v2:172.21.15.6:6802/2257685015,v1:172.21.15.6:6803/2257685015] exists,up 3bdfaa34-bc10-4cec-b5c2-86e05d444ffd', 'osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6824/2399951648,v1:172.21.15.50:6825/2399951648] [v2:172.21.15.50:6826/2399951648,v1:172.21.15.50:6827/2399951648] exists,up 5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2', 'osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6824/1225429423,v1:172.21.15.169:6825/1225429423] [v2:172.21.15.169:6826/1225429423,v1:172.21.15.169:6827/1225429423] exists,up ddd3edea-260a-4ec9-8533-841ed933fdb8', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6818/3415295348,v1:172.21.15.181:6819/3415295348] [v2:172.21.15.181:6820/3415295348,v1:172.21.15.181:6821/3415295348] exists,up 7afc1a18-98c2-4a58-a09d-1b4f474d4dc4', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6824/2216439459,v1:172.21.15.6:6825/2216439459] [v2:172.21.15.6:6826/2216439459,v1:172.21.15.6:6827/2216439459] exists,up 3e45f80f-856b-4d3f-b002-255f16a14f6b', 'osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6816/1727176716,v1:172.21.15.50:6817/1727176716] [v2:172.21.15.50:6818/1727176716,v1:172.21.15.50:6819/1727176716] exists,up b415c423-24de-4d86-bec4-ea330364ff96', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6808/3604183526,v1:172.21.15.169:6809/3604183526] [v2:172.21.15.169:6810/3604183526,v1:172.21.15.169:6811/3604183526] exists,up e471e1cb-9c0b-4c99-8622-59eb7956770d', 'osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6826/3420013159,v1:172.21.15.181:6827/3420013159] [v2:172.21.15.181:6828/3420013159,v1:172.21.15.181:6829/3420013159] exists,up c2bf4255-7ce1-4673-bedf-fd055d841fac', 'osd.12 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.6:6816/2310487797,v1:172.21.15.6:6817/2310487797] [v2:172.21.15.6:6818/2310487797,v1:172.21.15.6:6819/2310487797] exists,up 4c16f3ee-f427-41d5-b0ba-9e676ddbe032', 'osd.13 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6800/1210437211,v1:172.21.15.50:6802/1210437211] [v2:172.21.15.50:6804/1210437211,v1:172.21.15.50:6806/1210437211] exists,up 0a4386aa-1a59-44c8-8d1e-7d43bf10de2e', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] [v2:172.21.15.169:6802/4170280864,v1:172.21.15.169:6803/4170280864] exists,up 4c54e35c-e226-4d8c-bad0-51dd1cd95fe0', 'osd.15 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] [v2:172.21.15.181:6804/2815528414,v1:172.21.15.181:6805/2815528414] exists,up 0281e750-f74a-4760-bbef-fbfa4127d06b'] 2024-09-17T16:01:27.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-17T16:01:27.257 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:01:27.257 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mon.a", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster": "ceph", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket_mode": "", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_client_required": "cephx, none", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_cluster_required": "cephx", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_debug": "true", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_required": "cephx", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_supported": "cephx", 2024-09-17T16:01:27.258 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio": "true", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_poll_ms": "250", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_reap_max": "16", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_async_discard_threads": "0", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_block_size": "4096", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio": "false", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_inflight_ios": "false", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_enable_discard": "false", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry": "3", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash": "0", 2024-09-17T16:01:27.259 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring": "false", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_hipri": "false", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_type": "", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_alloc_size": "1048576", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_allocator": "hybrid", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_buffered_io": "true", 2024-09-17T16:01:27.260 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_for_zeros": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_compact_log_sync": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_log_runway": "4194304", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_prefetch": "1048576", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_flush_size": "524288", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_log_runway": "1048576", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-17T16:01:27.261 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_sync_write": "false", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocation_from_file": "true", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocator": "hybrid", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_async_db_compaction": "true", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-17T16:01:27.262 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bdev_label_multi": "true", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_create": "true", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_create": "false", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_path": "", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_size": "0", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_path": "", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_preallocate_file": "false", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_size": "107374182400", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_create": "false", 2024-09-17T16:01:27.263 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_path": "", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_size": "100663296", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs": "true", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune": "true", 2024-09-17T16:01:27.264 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size": "0", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_type": "2q", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_clone_cow": "true", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-17T16:01:27.265 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_mode": "none", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_csum_type": "crc32c", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_freelist": "false", 2024-09-17T16:01:27.266 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_small_allocations": "0", 2024-09-17T16:01:27.267 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_read": "true", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_write": "false", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fail_eio": "false", 2024-09-17T16:01:27.268 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-17T16:01:27.269 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_ignore_data_csum": "false", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_op_age": "5.000000", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_alloc_size": "0", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size": "0", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-17T16:01:27.270 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_deferred_txc": "32", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size": "0", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_nid_prealloc": "1024", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_retry_disk_reads": "3", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cf": "true", 2024-09-17T16:01:27.271 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_mem": "512", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-17T16:01:27.272 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_tracing": "false", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-17T16:01:27.273 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_write_v2": "false", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_write_v2_random": "false", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_zero_block_detection": "false", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-17T16:01:27.274 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_signatures": "false", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_version": "2", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_signatures": "false", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_version": "2", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_signatures": "false", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_version": "2", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_sign_messages": "true", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "chdir": "", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "client_acl_type": "", 2024-09-17T16:01:27.275 INFO:teuthology.orchestra.run.smithi006.stdout: "client_asio_thread_count": "2", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_mid": "0.750000", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_size": "16384", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_caps_release_delay": "5", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_check_pool_perm": "true", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_force_sync_read": "false", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_getattr_caps": "false", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_inject_features": "", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_inject_tick_delay": "0", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_remount": "false", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_dirsize_rbytes": "true", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_force_lazyio": "false", 2024-09-17T16:01:27.276 INFO:teuthology.orchestra.run.smithi006.stdout: "client_fs": "", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_release_failure": "false", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_inline_size": "4096", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mds_namespace": "", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_metadata": "", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_gid": "-1", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_timeout": "300", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_uid": "-1", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mountpoint": "/", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_notify_timeout": "10", 2024-09-17T16:01:27.277 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc": "true", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty": "104857600", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_objects": "1000", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_size": "209715200", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_target_dirty": "8388608", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_permissions": "true", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota": "true", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota_df": "true", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_bytes": "0", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_periods": "4", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_min": "131072", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_reconnect_stale": "false", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_shutdown_timeout": "30", 2024-09-17T16:01:27.278 INFO:teuthology.orchestra.run.smithi006.stdout: "client_snapdir": ".snap", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "client_tick_interval": "1", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "client_trace": "", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "client_try_dentry_invalidate": "false", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_faked_inos": "false", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_random_mds": "false", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog": "false", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_port": "12201", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_monitors": "default=true", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog": "false", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_level": "info", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "-", 2024-09-17T16:01:27.279 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network": "", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network_interface": "", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_isal": "false", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_level": "5", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_winsize": "-15", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zstd_level": "1", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_op_num_threads": "6", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_stat_interval": "0", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_seastar_cpu_cores": "", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_seastar_num_threads": "0", 2024-09-17T16:01:27.280 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook": "", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook_timeout": "10", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "daemonize": "false", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "daos_pool": "tank", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_alienstore": "0/5", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_allow_any_pool_priority": "false", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok": "1/5", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok_assert_abort": "false", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asserts_on_shutdown": "true", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_auth": "20/20", 2024-09-17T16:01:27.281 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bdev": "1/3", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluefs": "1/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluestore": "1/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_buffer": "0/1", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ceph_dedup": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ceph_exporter": "1/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephfs_mirror": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephsqlite": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_client": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_compressor": "1/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_context": "0/1", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crimson_interrupt": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crush": "1/1", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crypto": "1/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cyanstore": "0/5", 2024-09-17T16:01:27.282 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_deliberately_leak_memory": "false", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_disable_randomized_ping": "false", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_dpdk": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_eventtrace": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filer": "0/1", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filestore": "1/3", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_finisher": "1/1", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_fuse": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeat_testing_span": "0", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeatmap": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_javaclient": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journal": "1/3", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journaler": "0/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_kstore": "1/5", 2024-09-17T16:01:27.283 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_lockdep": "0/1", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mclock": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_balancer": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_locker": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log_expire": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_migrator": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_quiesce": "3/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_memstore": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgr": "2/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgrc": "1/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mon": "20/20", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_monc": "0/10", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ms": "1/1", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_none": "0/5", 2024-09-17T16:01:27.284 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objclass": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objectcacher": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objecter": "0/1", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_optracker": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_osd": "1/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_paxos": "20/20", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_perfcounter": "1/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_prioritycache": "1/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rados": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_mirror": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_pwl": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_replay": "0/5", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_refs": "0/0", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_reserver": "1/1", 2024-09-17T16:01:27.285 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_access": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_datacache": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_dbstore": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_flight": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_notification": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_sync": "1/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rocksdb": "4/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_backref": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cache": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cleaner": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_device": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_epm": "0/5", 2024-09-17T16:01:27.286 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_journal": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_lba": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_odata": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_omap": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_onode": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_t": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_tm": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_striper": "0/1", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_test": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_throttle": "1/1", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_timer": "0/1", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_tp": "0/5", 2024-09-17T16:01:27.287 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_trace": "1/5", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "device_failure_prediction_mode": "none", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_graylog": "false", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_journald": "false", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_stderr": "true", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_syslog": "false", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event_tracing": "false", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_addr": "0.0.0.0", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_cert_file": "", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_http_port": "9926", 2024-09-17T16:01:27.288 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_key_file": "", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_prio_limit": "5", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sort_metrics": "true", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_stats_period": "5", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "fake_statfs_for_testing": "0", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "fatal_signal_handlers": "true", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_purge_ops": "10", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_truncate_ops": "128", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_apply_finisher_threads": "1", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_blackhole": "false", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_clone_range": "true", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_snap": "true", 2024-09-17T16:01:27.289 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_caller_concurrency": "10", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_collect_device_partition_information": "true", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_commit_timeout": "600.000000", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_inject_read_err": "false", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_omap_check": "false", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_verify_split": "false", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_dump_file": "", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fadvise": "true", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fail_eio": "true", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_shards": "16", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_size": "128", 2024-09-17T16:01:27.290 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap": "false", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap_threshold": "4096", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_inject_stall": "0", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_parallel": "false", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_trailing": "false", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_writeahead": "false", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_kill_at": "0", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs": "0", 2024-09-17T16:01:27.291 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size": "0", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_merge_threshold": "-10", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_odsync_write": "false", 2024-09-17T16:01:27.292 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend": "rocksdb", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend_path": "", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_timeout": "60", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_threads": "2", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_punch_hole": "false", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-17T16:01:27.293 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_ops": "50", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_seek_data_hole": "false", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc": "false", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_splice": "false", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_multiple": "2", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_rand_factor": "20", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_update_to": "1000", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-17T16:01:27.294 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_enable": "true", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_xfs_extsize": "false", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_zfs_snap": "false", 2024-09-17T16:01:27.295 INFO:teuthology.orchestra.run.smithi006.stdout: "fio_dir": "/tmp/fio", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "4345ca04-2628-4bef-a537-6837268b96c9", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_allow_other": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_atomic_o_trunc": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_big_writes": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_debug": "false", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_default_permissions": "false", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_disable_pagecache": "false", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_max_write": "0", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_multithreaded": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_require_active_mds": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_set_user_groups": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_move": "true", 2024-09-17T16:01:27.296 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_read": "true", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_write": "true", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_use_invalidate_cb": "true", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_target_name": "ceph", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_file": "", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_inject_failure": "0", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_interval": "5", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "host": "smithi006", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-17T16:01:27.297 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "inject_early_sigterm": "false", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "jaeger_agent_port": "6799", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "jaeger_tracing_enable": "false", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_aio": "true", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_align_min_size": "65536", 2024-09-17T16:01:27.298 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_align": "true", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_size": "4096", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_dio": "true", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_discard": "false", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_force_aio": "false", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_ignore_corruption": "false", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_bytes": "10485760", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_entries": "100", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_replay_from": "0", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_write_header_frequency": "0", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_zero_on_create": "false", 2024-09-17T16:01:27.299 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prefetch_periods": "10", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prezero_periods": "5", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_write_head_interval": "15", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "keyfile": "", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_backend": "rocksdb", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_default_stripe_size": "65536", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount": "false", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_bytes": "67108864", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_ops": "512", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_nid_prealloc": "1024", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_onode_map_size": "1024", 2024-09-17T16:01:27.300 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_submit_transaction": "false", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_transaction": "false", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "librados_thread_count": "2", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep": "false", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep_force_backtrace": "false", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_coarse_timestamps": "true", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_flush_on_exit": "false", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_host": "127.0.0.1", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_port": "12201", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_new": "1000", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_recent": "10000", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stderr_prefix": "", 2024-09-17T16:01:27.301 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stop_at_utilization": "0.970000", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": "true", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_graylog": "false", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_journald": "false", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_stderr": "false", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_syslog": "false", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "max_rotating_auth_attempts": "10", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_action_on_write_error": "1", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_alternate_name_max": "8192", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_asio_thread_count": "2", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_export_pin": "true", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_dirs": "true", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-17T16:01:27.302 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_interval": "5", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_interval": "10", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max": "-1", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max_until": "-1", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_merge_size": "50", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_midchunk": "0.300000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_start": "0.200000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_minchunk": "0.001000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_mode": "0", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_max": "1.200000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_min": "0.800000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_overload_epochs": "2", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-17T16:01:27.303 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_bits": "3", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_size": "10000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_target_decay": "10.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_grace": "15.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_interval": "4.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_mid": "0.700000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_delay": "0", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-17T16:01:27.304 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_release_free_interval": "10", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_reservation": "0.050000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_interval": "1", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_threshold": "262144", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_prealloc_inos": "1000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_connect_bootstrapping": "false", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_damage_table_max_entries": "10000", 2024-09-17T16:01:27.305 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_auth_pins": "false", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_frag": "false", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_scatterstat": "false", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_subtrees": "false", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_decay_halflife": "5.000000", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_default_dir_hash": "2", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_defer_session_stale": "true", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_deny_all_reconnect": "false", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_keys_per_op": "16384", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_commit_size": "10", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_entries": "0", 2024-09-17T16:01:27.306 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_prefetch": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_on_map": "false", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_early_reply": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enable_op_tracker": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enforce_unique_name": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random": "true", 2024-09-17T16:01:27.307 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_summarize_threshold": "10", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_health_dummy": "false", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-17T16:01:27.308 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_migrator_session_race": "false", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_join_fs": "", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_journal_format": "1", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_create_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_dirfrag_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_export_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_import_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_expire_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_replay_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_link_at": "0", 2024-09-17T16:01:27.309 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_mdstable_at": "0", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_openc_at": "0", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_rename_at": "0", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_shutdown_at": "0", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_event_large_threshold": "524288", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_events_per_segment": "1024", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_events": "-1", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_segments": "128", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_pause": "false", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_segment_size": "0", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-17T16:01:27.310 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_threshold": "128", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_warn_factor": "2.000000", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_caps_per_client": "1048576", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_flushes": "100000", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_requests": "100000", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_export_size": "20971520", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_file_recover": "32", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_files": "64", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops": "8192", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_snaps_per_dir": "100", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_metrics_update_interval": "2", 2024-09-17T16:01:27.311 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_per_client": "100", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_working_set": "10000", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_numa_node": "-1", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_complaint_time": "30.000000", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_duration": "600", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_size": "20", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_size": "20", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_log_threshold": "5", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_grace": "15", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_interval": "5", 2024-09-17T16:01:27.312 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_caps": "30000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_threshold": "262144", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_interval": "1.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_gid": "0", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_uid": "0", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-17T16:01:27.313 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_scrub_stats_review_period": "1", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_shutdown_check": "0", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_skip_ino": "0", 2024-09-17T16:01:27.314 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_max_uid": "4294967294", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_min_uid": "0", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_rstat": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_standby_replay_damaged": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_symlink_recovery": "true", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_exports": "0", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_fragments": "0", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_tick_interval": "5.000000", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_valgrind_exit": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_backtrace": "1", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_scatter": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_sessions": "false", 2024-09-17T16:01:27.315 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool_debug": "false", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_device_bytes": "1073741824", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_set": "false", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_size": "65536", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_bytes": "134217728", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_messages": "512", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_disabled_modules": "", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_enable_op_tracker": "true", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-17T16:01:27.316 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_max_pg_creating": "1024", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_max_pg_num_change": "128", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_bytes": "134217728", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_messages": "128", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_bytes": "134217728", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_messages": "128", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_history_duration": "600", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_history_size": "20", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_op_log_threshold": "5", 2024-09-17T16:01:27.317 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_bytes": "536870912", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_messages": "8192", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_pool": "true", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_standby_modules": "true", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_period": "5", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_threshold": "5", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_tick_period": "2", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_delete": "true", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_size_one": "true", 2024-09-17T16:01:27.318 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_auth_validate_all_caps": "true", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_bytes": "104857600", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_directed_command_retry": "5", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_parallel": "3", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_log_interval": "1.000000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_interval": "10.000000", 2024-09-17T16:01:27.319 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_target_rank": "-1", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_level": "debug", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_file": "true", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_journald": "false", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-17T16:01:27.320 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_bootstrap": "false", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_start": "false", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_trim": "true", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cpu_threads": "4", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_crush_min_required_version": "hammer", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_daemon_bytes": "419430400", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_crit": "5", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_warn": "5", 2024-09-17T16:01:27.321 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_size_warn": "16106127360", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_json": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_transactions": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_extra_checks": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_reef": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_squid": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-17T16:01:27.322 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_down_mkfs_grace": "120", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_down_uptime_grace": "60", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_default_strategy": "1", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_timeout": "5.000000", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_divisor": "2", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_enable_op_tracker": "true", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_fake_pool_delete": "false", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_force_quorum_join": "false", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_globalid_prealloc": "10000", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_detail_to_clog": "false", 2024-09-17T16:01:27.323 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_log_update_period": "5", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_max_detail": "50", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog": "true", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_interval": "600", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host": "172.21.15.6,172.21.15.50,172.21.15.169", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host_override": "", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_initial_members": "", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-17T16:01:27.324 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease": "5.000000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_full_interval": "50", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max": "10000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max_summary": "50", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_epochs": "500", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_osd": "10000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pg_per_osd": "10000", 2024-09-17T16:01:27.325 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pool_pg_num": "65536", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_force_trim_to": "0", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_skip_sanity": "false", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_autotune": "true", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_target": "2147483648", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_beacon_grace": "30", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_digest_period": "5", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_inactive_grace": "60", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-17T16:01:27.326 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_min_osdmap_epochs": "500", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_complaint_time": "30", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_duration": "600", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_size": "20", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_size": "20", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_log_threshold": "5", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-17T16:01:27.327 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_in": "false", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size": "500", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_interval": "600", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_force_trim_to": "0", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-17T16:01:27.328 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_down_reporters": "2", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_report_timeout": "900", 2024-09-17T16:01:27.329 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_stuck_threshold": "60", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-17T16:01:27.330 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_probe_timeout": "2.000000", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_change": "0.050000", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_osds": "4", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-17T16:01:27.331 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_interval": "300", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_max_keys": "100", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_timeout": "300", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_session_timeout": "300", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_smart_report_timeout": "5", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stat_smooth_intervals": "6", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_min_size": "2", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_size": "4", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_debug": "false", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-17T16:01:27.332 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_provider_kill_at": "0", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_requester_kill_at": "0", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_timeout": "60.000000", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_target_pg_per_osd": "100", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_tick_interval": "5", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_interval": "300.000000", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_use_min_delay_socket": "false", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_older_version_delay": "604800", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-17T16:01:27.333 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_misplaced": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_older_version": "true", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-17T16:01:27.334 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": "", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_tracing_enabled": "false", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_op_threads": "3", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_cm": "false", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_device_name": "", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_dscp": "96", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-17T16:01:27.335 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_local_gid": "", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_port_num": "1", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_sl": "3", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_support_srq": "true", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_type": "ib", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_reap_threshold": "5", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_before_connect": "false", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv4": "true", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv6": "false", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr1": "true", 2024-09-17T16:01:27.336 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr2": "true", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_max": "7568", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_min": "6800", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_count": "3", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_delay": "5", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_client": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mds": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mgr": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mon": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_osd": "false", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_client_mode": "crc secure", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_mode": "crc secure", 2024-09-17T16:01:27.337 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_type": "", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_compress_secure": "false", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_idle_timeout": "900", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_ready_timeout": "10", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_data": "true", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_header": "true", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bad_msg": "false", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bug": "true", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_old_message": "true", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_skipped_message": "false", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_coremask": "0xF", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-17T16:01:27.338 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_enable_tso": "true", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hugepages": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_lro": "true", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_memory_channel": "4", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_pmd": "", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_port_id": "0", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_on_send": "false", 2024-09-17T16:01:27.339 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_initial_backoff": "0.200000", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_max": "1.000000", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_type": "osd", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_network_congestion": "0", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_socket_failures": "2500", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_learn_addr_from_peer": "true", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_accept_failures": "4", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_backoff": "15.000000", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_client_mode": "secure crc", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_service_mode": "secure crc", 2024-09-17T16:01:27.340 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_min_size": "1024", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_mode": "none", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_min_cost": "65536", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_public_type": "", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_service_mode": "crc secure", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_listen_backlog": "512", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_nodelay": "true", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_rcvbuf": "0", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_type": "async+posix", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "no_config_file": "false", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-17T16:01:27.341 INFO:teuthology.orchestra.run.smithi006.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_completion_locks_per_session": "32", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_ops": "1024", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_tick_interval": "5.000000", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_timeout": "10.000000", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objectstore_blackhole": "false", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "openssl_engine_opts": "", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_delay_time": "5.000000", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_hist_halflife": "1000", 2024-09-17T16:01:27.342 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_low_ops": "2", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_ops": "4", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_slop": "0.020000", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_asio_thread_count": "2", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_async_recovery_min_cost": "100", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_max": "512", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_min": "64", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_degraded": "false", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_peering": "false", 2024-09-17T16:01:27.343 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_unfound": "true", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_beacon_report_interval": "300", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_duration": "30", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_max_block_size": "67108864", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_blkin_trace_all": "false", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_for_log_corruption": "false", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T16:01:27.344 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_update_on_start": "true", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_cap": "256", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_size_cap": "524288000", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_op_priority": "63", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_watch_timeout": "30", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_command_max_records": "256", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_compact_on_start": "false", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_on_start": "true", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_weight_set": "true", 2024-09-17T16:01:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_feed_pullee": "-1", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_misdirected_ops": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_acting_change": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_purge_strays": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_op_order": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-17T16:01:27.346 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_shutdown": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_trim_objects": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_snaps": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_keys": "1024", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-17T16:01:27.347 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_stride": "524288", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_notify_timeout": "30", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep": "0.000000", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_discard_disconnected_ops": "true", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_ec_partial_reads": "true", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_enable_op_tracker": "true", 2024-09-17T16:01:27.348 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_info": "true", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown": "true", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-17T16:01:27.349 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_function_tracing": "false", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_grace": "20", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_interval": "6", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_peers": "10", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_size": "2000", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_stale": "600", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_max_size": "100000", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_min_size": "1000", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-17T16:01:27.350 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_size": "5120", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_kill_backfill_at": "0", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_cache_size": "50", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_dedup": "true", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max": "40", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_share_max_epochs": "40", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_name_len": "100", 2024-09-17T16:01:27.351 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_size": "0", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_backfills": "1", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_count": "5", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_period": "600", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_name_len": "2048", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_namespace_len": "256", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_size": "134217728", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_blocked_by": "16", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_log_entries": "10000", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pgls": "1024", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_cost": "8388608", 2024-09-17T16:01:27.352 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_objects": "10", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_scrubs": "3", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_trimming_pgs": "2", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_op_reply_len": "64", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_size": "90", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-17T16:01:27.353 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_profile": "balanced", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_base": "805306368", 2024-09-17T16:01:27.354 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_min": "134217728", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target": "4294967296", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_autotune": "false", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_pg_log_entries": "250", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_recovery_priority": "0", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_interval": "5", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_cache_shards": "32", 2024-09-17T16:01:27.355 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_op_tracker_shard": "32", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_auto_affinity": "true", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_node": "-1", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_prefer_iface": "true", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objecter_finishers": "1", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore": "bluestore", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_fuse": "false", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_tracing": "false", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_complaint_time": "30.000000", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_duration": "600", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_size": "20", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_size": "20", 2024-09-17T16:01:27.356 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_log_threshold": "5", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards": "0", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_hdd": "5", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_ssd": "8", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_min_cost": "65536", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue_cut_off": "high", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-17T16:01:27.357 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_timeout": "15", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_open_classes_on_start": "true", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_os_flags": "0", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_peering_op_priority": "255", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_cost": "1048576", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_priority": "5", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_max": "10000", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_min": "100", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-17T16:01:27.358 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_crimson": "false", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-17T16:01:27.359 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flags": "0", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_min_size": "0", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_num": "32", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pgp_num": "0", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_read_ratio": "70", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_size": "2", 2024-09-17T16:01:27.360 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_type": "replicated", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_push_per_object_cost": "1000", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap": "true", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_cost": "20971520", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active": "0", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-17T16:01:27.361 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_single_start": "1", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_priority": "3", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_priority": "5", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep": "0.000000", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_repair_during_recovery": "false", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_requested_scrub_priority": "120", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-17T16:01:27.362 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair": "false", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_hour": "0", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_week_day": "0", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_max": "15", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_min": "5", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_cost": "52428800", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_during_recovery": "false", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_hour": "0", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_week_day": "0", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_event_cost": "4096", 2024-09-17T16:01:27.363 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_invalid_stats": "true", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_preemptions": "5", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_priority": "5", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_retry_delay": "30", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_retry_trimming": "10", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_sleep": "0.000000", 2024-09-17T16:01:27.364 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_skip_data_digest": "false", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_smart_report_timeout": "5", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_cost": "1048576", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_priority": "5", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-17T16:01:27.365 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_transaction_size": "30", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tracing": "false", 2024-09-17T16:01:27.366 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_use_stale_snap": "false", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "osdc_blkin_trace_all": "false", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_kill_at": "0", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_max_join_drift": "10", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min": "500", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min_wait": "0.050000", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_propose_interval": "1.000000", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max": "500", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_min": "250", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_stash_full_interval": "25", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_max": "500", 2024-09-17T16:01:27.367 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_min": "250", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "perf": "true", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "-", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrv": "", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "public_bind_addr": "-", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network": "", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network_interface": "", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_busy_polling": "false", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_enabled": "false", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_session_max_number": "256", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_mon_op_timeout": "0", 2024-09-17T16:01:27.368 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_osd_op_timeout": "0", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_tracing": "false", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_atime_update_interval": "60", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_parent_reads": "false", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_snap_reads": "false", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blkin_trace_all": "false", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache": "true", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-17T16:01:27.369 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_policy": "writearound", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_size": "33554432", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_clone_copy_on_read": "false", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_compression_hint": "none", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_concurrent_management_ops": "10", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_clone_format": "auto", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_data_pool": "", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_features": "61", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_format": "2", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_map_options": "", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_order": "22", 2024-09-17T16:01:27.370 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_pool": "rbd", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_count": "0", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_unit": "0", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_enable_alloc_hint": "true", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler": "simple", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-17T16:01:27.371 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_order": "24", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_pool": "", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_splay_width": "4", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_parent_reads": "false", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_snap_reads": "false", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-17T16:01:27.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-17T16:01:27.373 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-17T16:01:27.374 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mtime_update_interval": "60", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_non_blocking_aio": "true", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_thread_timeout": "60", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_threads": "1", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_parent_cache_enabled": "false", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-17T16:01:27.375 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_plugins": "", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_limit": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_exclude_ops": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_limit": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-17T16:01:27.376 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_read_from_replica_policy": "default", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-17T16:01:27.377 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_skip_partial_discard": "true", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_tracing": "false", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_names": "true", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_pool": "true", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_base_url": "", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_log_level": "", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_acl_grants_max_num": "100", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_admin_entry": "admin", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_asio_assert_yielding": "false", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_backend_store": "rados", 2024-09-17T16:01:27.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_barbican_url": "", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_beast_enable_async": "true", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_counters_cache": "false", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_enabled": "true", 2024-09-17T16:01:27.379 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_expiry_interval": "900", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_lru_size": "25000", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_config_store": "rados", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_content_length_compat": "false", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress": "true", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cors_rules_max_num": "100", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cross_domain_policy": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_addr": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-17T16:01:27.380 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_password": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_username": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_require_ssl": "true", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-17T16:01:27.381 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_addr": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_auth": "token", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_namespace": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_prefix": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-17T16:01:27.382 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_token_file": "", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_buffersize": "524288", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_time": "300", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-17T16:01:27.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_changes_size": "1000", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_num_shards": "128", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-17T16:01:27.384 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_window": "30", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-17T16:01:27.385 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_disable_s3select": "false", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_name": "", 2024-09-17T16:01:27.386 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_s3website_name": "", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dynamic_resharding": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_gc_threads": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_lc_threads": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_mdsearch": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_ops_log": "false", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_quota_threads": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_static_website": "false", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_usage_log": "false", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enforce_swift_acls": "true", 2024-09-17T16:01:27.387 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_exit_timeout_secs": "120", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_expose_bucket": "false", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_extended_http_attrs": "", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_filter": "none", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontends": "beast port=7480", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred": "50", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_objs": "32", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-17T16:01:27.388 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_period": "3600", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_init_timeout": "300", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_connection_idle": "300", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-17T16:01:27.389 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_domain": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password_path": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_project": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_tenant": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token_path": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_user": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_domain": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_password": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_project": "", 2024-09-17T16:01:27.390 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_user": "", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_url": "", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_debug_interval": "-1", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_lock_max_time": "90", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_objs": "32", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_rules": "1000", 2024-09-17T16:01:27.391 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_worker": "3", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_thread_delay": "0", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_dnattr": "uid", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchfilter": "", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_uri": "", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-17T16:01:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_http_headers": "", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name_utc": "false", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_name_len": "0", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_size": "0", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_chunk_size": "4194304", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-17T16:01:27.393 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_listing_results": "5000", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_notify_retries": "10", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_param_size": "1048576", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_size": "5368709120", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_slo_entries": "1000", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_log_max_shards": "64", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mp_lock_max_time": "600", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-17T16:01:27.394 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_max_gc": "300", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_async_rados_threads": "32", 2024-09-17T16:01:27.395 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_control_oids": "8", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_numa_node": "-1", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_chunk_size": "100", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_gc_interval": "600", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_timeout": "600", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_tracing": "false", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_token": "", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_url": "", 2024-09-17T16:01:27.396 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_verify_ssl": "true", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_rados": "false", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_socket_path": "", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval": "2.000000", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-17T16:01:27.397 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_lanes": "3", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_partitions": "3", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_continue": "true", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_tracing": "false", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm": "", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_id": "", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-17T16:01:27.398 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region": "", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_request_uri": "", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_batch_size": "64", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_max_aio": "128", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_num_logs": "16", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_thread_interval": "600", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-17T16:01:27.399 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_resolve_cname": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_run_sync_thread": "true", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_scheduler_type": "throttler", 2024-09-17T16:01:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_script_uri": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_service_provider_name": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_id": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_secret": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_entry": "sts", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_key": "sts", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_min_session_duration": "900", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_token_introspection_url": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_account_in_url": "false", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_entry": "auth", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_url": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_custom_header": "", 2024-09-17T16:01:27.401 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_need_stats": "true", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_tenant_name": "", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_token_expiration": "86400", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url": "", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url_prefix": "swift", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_lease_period": "120", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-17T16:01:27.402 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_thread_pool_size": "512", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_comment": "", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_createby": "", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_encoding": "", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_flag": "false", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-17T16:01:27.403 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_origin": "", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_tracker": "", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_trust_forwarded_https": "false", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_shards": "32", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_user_shards": "1", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_use_opa_authz": "false", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_counters_cache": "false", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_max_buckets": "1000", 2024-09-17T16:01:27.404 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_policies_max_num": "100", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_unique_email": "true", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_verify_ssl": "true", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone": "", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_id": "", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup": "", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_id": "", 2024-09-17T16:01:27.405 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgwlc_auto_session_clear": "true", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_block_size": "4096", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_size": "536870912", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-17T16:01:27.406 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_index_type": "binary_search", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_partition_filters": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_perf": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-17T16:01:27.407 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "run_dir": "/var/run/ceph", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_block_create": "true", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_cache_lru_size": "67108864", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_max_object_size": "16777216", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_device_size": "53687091200", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_full_integrity_check": "false", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_capacity": "16", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-17T16:01:27.408 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_main_device_type": "SSD", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_segment_size": "67108864", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "set_keepcaps": "false", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "setgroup": "", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser": "", 2024-09-17T16:01:27.409 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser_match_path": "", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "thp": "false", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_default_timeout": "60", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "throttler_perf_counter": "true", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "tmp_dir": "/tmp", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "uadk_compressor_enabled": "false", 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-17T16:01:27.410 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:01:27.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-17T16:01:27.538 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-09-17T16:01:27.538 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:01:27.553 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-09-17T16:01:27.553 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:01:27.568 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-09-17T16:01:27.568 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:01:27.579 DEBUG:teuthology.orchestra.run.smithi181:> type ceph-objectstore-tool 2024-09-17T16:01:27.597 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T16:01:27.598 DEBUG:teuthology.orchestra.run.smithi006:> type ceph-objectstore-tool 2024-09-17T16:01:27.612 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T16:01:27.613 DEBUG:teuthology.orchestra.run.smithi169:> type ceph-objectstore-tool 2024-09-17T16:01:27.631 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T16:01:27.631 DEBUG:teuthology.orchestra.run.smithi050:> type ceph-objectstore-tool 2024-09-17T16:01:27.650 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T16:01:27.650 INFO:teuthology.run_tasks:Running task rados... 2024-09-17T16:01:27.661 INFO:tasks.rados:Beginning rados... 2024-09-17T16:01:27.662 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-17T16:01:27.662 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T16:01:27.662 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-17T16:01:27.662 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-17T16:01:27.670 INFO:tasks.rados:joining rados 2024-09-17T16:01:27.670 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-17T16:01:27.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-17T16:01:27.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:01:27.670 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-17T16:01:27.671 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-17T16:01:27.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:27.671 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-17T16:01:27.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:01:27.672 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-17T16:01:27.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:01:27.672 INFO:tasks.rados:clients are ['client.0'] 2024-09-17T16:01:27.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-17T16:01:27.673 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:27.670+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:27.772 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:27.770+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:27.789 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:01:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:01:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:01:27.789 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:01:27.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:01:27.818 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:27.818 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:27.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:27.856 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:27.873 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:27.871+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:27.878 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:27.879 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:27.900 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:27.900 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:27.914 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:27.914 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:27.921 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:27.922 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:27.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:01:27.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:27.944 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:27.966 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:27.966 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:27.974 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:27.971+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:27.988 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:27.988 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:28.010 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:28.011 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:28.032 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:28.033 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:28.053 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.053 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.054 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:28.054 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:28.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:01:28.074 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:28.072+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:28.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:28.076 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:28.097 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:28.097 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:28.119 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:28.119 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:28.125 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:28.123+0000 7fa70ffff640 0 -- 172.21.15.6:0/1841175954 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7fa710221930 msgr2=0x7fa710223d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:28.175 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:28.172+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:28.183 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.183 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:01:28.276 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:28.274+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:28.339 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.339 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:01:28.391 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:28.389+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:28.454 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.455 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:01:28.504 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:28.502+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:28.578 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:28.577+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:28.611 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.611 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:01:28.680 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:28.678+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:28.726 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.726 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:01:28.780 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:28.779+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:28.817 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:01:28.817 INFO:teuthology.orchestra.run.smithi006.stderr:load: shec 2024-09-17T16:01:28.833 INFO:tasks.rados:starting run 0 out of 1 2024-09-17T16:01:28.833 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-17T16:01:28.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-17T16:01:28.850 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:28.850 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:28.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:01:28.881 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:28.879+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:28.982 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:28.981+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:29.043 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.043 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:01:29.061 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:29.061 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:29.083 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:29.081+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:29.084 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:29.084 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:29.168 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.168 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:01:29.221 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:29.219+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:29.285 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.285 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.285 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:29.282+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:29.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:01:29.385 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:29.383+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:29.414 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.414 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:01:29.486 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:29.485+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:29.531 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.531 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:01:29.588 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:29.585+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:29.688 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.689 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.689 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:29.687+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:29.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:01:29.790 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:29.788+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:29.845 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.846 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:01:29.891 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:29.889+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:29.962 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:29.962 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:29.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:01:30.014 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:30.012+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:30.077 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.077 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:01:30.093 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:30.091+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:30.194 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:30.191+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:30.233 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.233 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:01:30.295 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:30.292+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:30.349 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.349 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:01:30.396 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:30.394+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:30.498 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:30.496+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:30.506 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.506 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:01:30.598 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:30.595+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:30.621 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.622 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:01:30.701 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:30.699+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:30.784 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.784 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:01:30.800 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:30.798+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:30.827 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' created 2024-09-17T16:01:30.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:01:30.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-17T16:01:30.899 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:30.899 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:30.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:01:30.932 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:30.930+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:31.002 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:31.000+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:31.023 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:31.023 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:31.032 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:01:31.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:01:31.104 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:31.101+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:31.133 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:31.134 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:31.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:01:31.204 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:31.202+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:31.234 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:31.235 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:31.247 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:31.247 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:31.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:01:31.257 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:31.258 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:31.278 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:31.278 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:31.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:31.300 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:31.320 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:31.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:31.342 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:31.340+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:31.342 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:31.342 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:31.363 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:31.363 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:31.384 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:31.385 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:31.401 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:31.401 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:31.401 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:31.401 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:31.406 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:31.404+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:31.407 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:31.407 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:31.409 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:01:31.428 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:31.429 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:31.450 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:31.450 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:31.471 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:31.472 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:31.491 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:31.492 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:31.507 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:31.504+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:31.513 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:31.513 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:31.517+0000 7f152a7fc640 0 -- 172.21.15.6:0/3662237900 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7f1534221750 msgr2=0x7f1534223b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:31.540 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:31.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:31.554 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:31.555 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:31.555 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:31.555 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:31.555 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:31.563 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:01:31.566 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:31.607 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:31.605+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:31.666 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:31.666 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:31.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:31.666 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [] 2024-09-17T16:01:31.666 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:31.675 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:01:31.709 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:31.706+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:31.778 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:31.778 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:31.778 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:31.778 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:31.786 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:01:31.831 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:31.829+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:31.838 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:01:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-17T16:01:31.895 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:31.895 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:31.895 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:31.895 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:31.895 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:31.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:01:31.910 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:31.908+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:32.011 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:32.009+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:32.050 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:32.050 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:32.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:32.051 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [] 2024-09-17T16:01:32.051 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:32.059 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:01:32.111 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:32.109+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:32.162 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:32.162 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:32.162 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:32.162 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:32.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:01:32.212 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:32.209+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:32.312 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:32.313 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:32.313 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:32.313 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:32.313 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:32.313 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:32.311+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:32.321 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:01:32.414 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:32.411+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:32.463 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:32.463 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:32.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:32.463 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [] 2024-09-17T16:01:32.463 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:32.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:01:32.515 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:32.512+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:32.589 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:32.589 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:32.589 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:32.589 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:32.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:01:32.615 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:32.613+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:32.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-17T16:01:32.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-17T16:01:32.701 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:32.701 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:32.701 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:01:32.702 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:01:32.702 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:32.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:01:32.716 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:32.714+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:32.816 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:32.816 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:01:32.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:01:32.817 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-09-17T16:01:32.817 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:32.818 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:32.816+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:32.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:01:32.861 DEBUG:teuthology.orchestra.run.smithi181:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight read -> 100 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight delete -> 50 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight snap_create -> 50 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight snap_remove -> 50 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight rollback -> 50 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight setattr -> 25 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight rmattr -> 25 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight copy_from -> 50 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:adding op weight append -> 100 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev) 2024-09-17T16:01:32.893 INFO:tasks.rados.rados.0.smithi181.stdout:Configuration: 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Number of operations: 400 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Number of objects: 50 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Max in flight operations: 16 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Object size (in bytes): 4000000 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Write stride min: 400000 2024-09-17T16:01:32.894 INFO:tasks.rados.rados.0.smithi181.stdout: Write stride max: 800000 2024-09-17T16:01:32.919 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:32.917+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:32.927 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:32.927 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:32.928 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:32.928 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:32.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-17T16:01:33.021 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:33.019+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:33.081 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.081 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:33.081 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:01:33.081 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:01:33.081 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-17T16:01:33.133 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:33.131+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:33.192 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.192 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:01:33.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:01:33.193 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [] 2024-09-17T16:01:33.193 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:01:33.222 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:33.221+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:33.324 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:33.322+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:33.344 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.344 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:33.344 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:33.344 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:01:33.425 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:33.423+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:33.457 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.457 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:33.457 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:01:33.457 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:01:33.457 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:01:33.526 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:33.524+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:33.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:01:33.610 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.857967854000002, 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19491629299999999, 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748958+0000", 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748966+0000", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5870000000000004e-06 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748979+0000", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748988+0000", 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:01:33.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749052+0000", 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4618999999999999e-05 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749064+0000", 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.939947+0000", 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19088328399999999 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940079+0000", 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013157500000000001 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940245+0000", 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016577900000000001 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943345+0000", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031004499999999998 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943806+0000", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046112200000000001 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943843+0000", 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6578000000000001e-05 2024-09-17T16:01:33.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943873+0000", 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [cmpxattr striper.excl (25) op 1 mode 1 in=12b,setxattr striper.excl in=12b,call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.658882527999999, 2024-09-17T16:01:33.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83632086699999997, 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:01:33.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948043+0000", 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948048+0000", 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948060+0000", 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1761e-05 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948066+0000", 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5350000000000001e-06 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948111+0000", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4783000000000001e-05 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948190+0000", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948399+0000", 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000209292 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948922+0000", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052251800000000003 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784264+0000", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83534246599999995 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784309+0000", 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4660000000000003e-05 2024-09-17T16:01:33.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784363+0000", 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4095000000000003e-05 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.4219.0:8 1.0 e24/23 ondisk, result = 0)", 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:01:33.623 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.657760332999999, 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83521008600000002, 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949165+0000", 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949169+0000", 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9339999999999999e-06 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784135+0000", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83496651200000005 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784151+0000", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784250+0000", 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8222999999999998e-05 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784260+0000", 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0265000000000001e-05 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784374+0000", 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114373 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.627 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:47 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.922987+0000", 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683937686, 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097752640000000005, 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 47 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.922987+0000", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.922987+0000", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.922987+0000", 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8799999999999998e-07 2024-09-17T16:01:33.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923056+0000", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8612e-05 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923063+0000", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1760000000000004e-06 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923069+0000", 2024-09-17T16:01:33.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7710000000000003e-06 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928059+0000", 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049902970000000003 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928127+0000", 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7733999999999999e-05 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928274+0000", 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014687200000000001 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931422+0000", 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003148615 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932678+0000", 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012561499999999999 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932727+0000", 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8643999999999997e-05 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932762+0000", 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5104999999999997e-05 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.633 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.923081+0000", 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683843857999999, 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011605257000000001, 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 48 2024-09-17T16:01:33.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923081+0000", 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923081+0000", 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923081+0000", 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923086+0000", 2024-09-17T16:01:33.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5120000000000002e-06 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923092+0000", 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3369999999999996e-06 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923097+0000", 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6330000000000004e-06 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928591+0000", 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054942639999999996 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928670+0000", 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9198999999999998e-05 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.928790+0000", 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120103 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931427+0000", 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026361639999999999 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934605+0000", 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003178808 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934653+0000", 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7358000000000003e-05 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934686+0000", 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3198000000000003e-05 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.640 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.923111+0000", 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683813934, 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011685360000000001, 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 49 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923111+0000", 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923111+0000", 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923111+0000", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1500000000000004e-07 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923116+0000", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0039999999999999e-06 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923122+0000", 2024-09-17T16:01:33.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2559999999999999e-06 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923128+0000", 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1440000000000002e-06 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.929127+0000", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059993900000000003 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.929215+0000", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8162000000000001e-05 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.929346+0000", 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013039400000000001 2024-09-17T16:01:33.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931882+0000", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025367609999999998 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934734+0000", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028514930000000001 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934769+0000", 2024-09-17T16:01:33.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4963999999999999e-05 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934796+0000", 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7177000000000001e-05 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.647 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.923140+0000", 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683784641000001, 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011795831999999999, 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 50 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923140+0000", 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923140+0000", 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923140+0000", 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7899999999999998e-07 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923225+0000", 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4171000000000004e-05 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923233+0000", 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1829999999999994e-06 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923239+0000", 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6470000000000001e-06 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.930262+0000", 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0070227409999999999 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.930347+0000", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290999999999996e-05 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.930488+0000", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014144900000000001 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932170+0000", 2024-09-17T16:01:33.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001681987 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934852+0000", 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026818710000000002 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934900+0000", 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8464999999999998e-05 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934936+0000", 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5447999999999997e-05 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.923252+0000", 2024-09-17T16:01:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683672477, 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011770821000000001, 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 51 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923252+0000", 2024-09-17T16:01:33.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923252+0000", 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923253+0000", 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9999999999999997e-07 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923257+0000", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3370000000000001e-06 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923264+0000", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8580000000000002e-06 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923271+0000", 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.703e-06 2024-09-17T16:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.930987+0000", 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077159840000000004 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931068+0000", 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1279000000000006e-05 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931188+0000", 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011993999999999999 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932642+0000", 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001454313 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.934973+0000", 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023308880000000001 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935001+0000", 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8092000000000001e-05 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935023+0000", 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1727000000000001e-05 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.660 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.923279+0000", 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.683645070000001, 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011863749999999999, 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 52 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923279+0000", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923279+0000", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923280+0000", 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:01:33.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923284+0000", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2119999999999997e-06 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923290+0000", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7899999999999996e-06 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.923295+0000", 2024-09-17T16:01:33.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9740000000000001e-06 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931601+0000", 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0083058730000000001 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931663+0000", 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1997999999999998e-05 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.931738+0000", 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5438000000000005e-05 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.932967+0000", 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001228839 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935059+0000", 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020919020000000001 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935110+0000", 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0556999999999997e-05 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935143+0000", 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3658000000000002e-05 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.666 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:56 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 262144~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:17.935542+0000", 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.671382775, 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0098540099999999999, 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 56 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935542+0000", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935542+0000", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935542+0000", 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2599999999999998e-07 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935597+0000", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5373e-05 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935602+0000", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7849999999999999e-06 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.935606+0000", 2024-09-17T16:01:33.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4130000000000002e-06 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.937204+0000", 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001597911 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.937283+0000", 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8955000000000006e-05 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.937417+0000", 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013430699999999999 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.937981+0000", 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000564183 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.945329+0000", 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073476219999999998 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.945362+0000", 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3250999999999997e-05 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.945396+0000", 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3884000000000003e-05 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:74 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.672 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.027289+0000", 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.579635335000001, 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097967060000000005, 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 74 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027289+0000", 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027289+0000", 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027289+0000", 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7399999999999999e-07 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027292+0000", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1059999999999998e-06 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027295+0000", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.304e-06 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027298+0000", 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.315e-06 2024-09-17T16:01:33.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.033461+0000", 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006162402 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.033536+0000", 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5572000000000001e-05 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.033681+0000", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014486800000000001 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.036283+0000", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026020010000000001 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037018+0000", 2024-09-17T16:01:33.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00073475799999999996 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037051+0000", 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2786000000000001e-05 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037086+0000", 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5320000000000001e-05 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.027302+0000", 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.579622937, 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0098393939999999996, 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 75 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.679 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027302+0000", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027302+0000", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027302+0000", 2024-09-17T16:01:33.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4699999999999998e-07 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027360+0000", 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8146999999999997e-05 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027366+0000", 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5400000000000003e-06 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027369+0000", 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4010000000000001e-06 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.033979+0000", 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066103409999999996 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.034030+0000", 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1205000000000002e-05 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.034107+0000", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6518000000000002e-05 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.036334+0000", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022270139999999998 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037090+0000", 2024-09-17T16:01:33.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00075554699999999997 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037121+0000", 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1613999999999999e-05 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037141+0000", 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.982e-05 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.027376+0000", 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.579548237999999, 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010248317, 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 76 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.685 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027376+0000", 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027376+0000", 2024-09-17T16:01:33.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027377+0000", 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1899999999999998e-07 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027380+0000", 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9289999999999998e-06 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027383+0000", 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3270000000000001e-06 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.027395+0000", 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2466000000000001e-05 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.036361+0000", 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0089652110000000007 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.036440+0000", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9674999999999999e-05 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.036557+0000", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000116563 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037146+0000", 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058942900000000004 2024-09-17T16:01:33.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037576+0000", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042955899999999998 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037604+0000", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8172000000000001e-05 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.037625+0000", 2024-09-17T16:01:33.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0567000000000001e-05 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.134425+0000", 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.472499067999999, 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012220227, 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.691 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 98 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134425+0000", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134425+0000", 2024-09-17T16:01:33.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134426+0000", 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8599999999999999e-07 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134431+0000", 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8450000000000002e-06 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134437+0000", 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1550000000000004e-06 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134442+0000", 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8559999999999998e-06 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140102+0000", 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056602390000000001 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140187+0000", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5002999999999999e-05 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140315+0000", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012855399999999999 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.141899+0000", 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015835599999999999 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146600+0000", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047012440000000003 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146628+0000", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7353000000000001e-05 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146646+0000", 2024-09-17T16:01:33.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8131999999999999e-05 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.134455+0000", 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.472469825999999, 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012293467000000001, 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.697 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 99 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134455+0000", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134455+0000", 2024-09-17T16:01:33.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134455+0000", 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134503+0000", 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7432999999999998e-05 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134510+0000", 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6790000000000005e-06 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134516+0000", 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0390000000000001e-06 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140623+0000", 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061062690000000001 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140691+0000", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8419999999999999e-05 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.140799+0000", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108167 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.141904+0000", 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011051959999999999 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146685+0000", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047803689999999996 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146722+0000", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7332000000000001e-05 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146748+0000", 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6033e-05 2024-09-17T16:01:33.702 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.134529+0000", 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.472395805, 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012287885, 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.703 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 100 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134529+0000", 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134529+0000", 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134529+0000", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3099999999999997e-07 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134533+0000", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0160000000000002e-06 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134541+0000", 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6320000000000008e-06 2024-09-17T16:01:33.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.134548+0000", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4270000000000002e-06 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.141376+0000", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0068271290000000004 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.141440+0000", 2024-09-17T16:01:33.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4122999999999996e-05 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.141518+0000", 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8182000000000005e-05 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.142111+0000", 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00059275699999999996 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146773+0000", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046621639999999999 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146797+0000", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4362000000000002e-05 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.146817+0000", 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9462000000000001e-05 2024-09-17T16:01:33.708 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.235871+0000", 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.371053707, 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010109589, 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.709 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 122 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235871+0000", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235871+0000", 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235871+0000", 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9499999999999998e-07 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235873+0000", 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.849e-06 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235876+0000", 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1360000000000001e-06 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235879+0000", 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3700000000000002e-06 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.242271+0000", 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063925229999999998 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.242321+0000", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0317e-05 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.242406+0000", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5012000000000005e-05 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243839+0000", 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014323739999999999 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.245896+0000", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002057199 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.245943+0000", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6513e-05 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.245980+0000", 2024-09-17T16:01:33.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7901000000000002e-05 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.235885+0000", 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.371039445999999, 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010213211999999999, 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.715 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 123 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235885+0000", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235885+0000", 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235885+0000", 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4900000000000002e-07 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235914+0000", 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9048e-05 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235918+0000", 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0799999999999999e-06 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235921+0000", 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.926e-06 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243110+0000", 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071885739999999997 2024-09-17T16:01:33.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243189+0000", 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8559000000000005e-05 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243313+0000", 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000124797 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.244320+0000", 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010066459999999999 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246025+0000", 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001704924 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246063+0000", 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8441e-05 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246098+0000", 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4968000000000001e-05 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.235929+0000", 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.370995887999999, 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010284951000000001, 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.721 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 124 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235929+0000", 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235929+0000", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235929+0000", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8200000000000001e-07 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235931+0000", 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3599999999999999e-06 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235939+0000", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5669999999999997e-06 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.235942+0000", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.942e-06 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243859+0000", 2024-09-17T16:01:33.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.007917693 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243918+0000", 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8099000000000001e-05 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.243990+0000", 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2347000000000004e-05 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.244459+0000", 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046867499999999999 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246134+0000", 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00167534 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246180+0000", 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5785000000000003e-05 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.246214+0000", 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3861e-05 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4296.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 14.847572120000001, 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.241196405, 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:33.727 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4296", 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.6:0/1428345084", 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759356+0000", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.484e-06 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759366+0000", 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9089999999999995e-06 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759375+0000", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759397+0000", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1824e-05 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759444+0000", 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6539000000000003e-05 2024-09-17T16:01:33.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759449+0000", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998167+0000", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23871730099999999 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998242+0000", 2024-09-17T16:01:33.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4892000000000001e-05 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0", 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998303+0000", 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0931999999999999e-05 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998708+0000", 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040511399999999998 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:01:33.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000499+0000", 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017911909999999999 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000526+0000", 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6798999999999999e-05 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000549+0000", 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3110000000000001e-05 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.733 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.734 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.734 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:33.734 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:33.734 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.735 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:33.735 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:33.732+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:33.735 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:33.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:01:33.747 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:33.747 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:33.769 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:33.769 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:33.790 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:33.790 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:33.811 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:33.832 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:33.836 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:33.834+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:33.839 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.839 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:33.839 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:33.839 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:01:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:1: write initial oid 1 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:1: seq_num 0 ranges {0=32768} 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:1: writing smithi18136759-1 from 0 to 32768 tid 1 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:2: write initial oid 2 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:2: seq_num 1 ranges {0=32768} 2024-09-17T16:01:33.850 INFO:tasks.rados.rados.0.smithi181.stdout:2: writing smithi18136759-2 from 0 to 32768 tid 1 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:3: write initial oid 3 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:3: seq_num 2 ranges {0=49152} 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:3: writing smithi18136759-3 from 0 to 49152 tid 1 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:4: write initial oid 4 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:4: seq_num 3 ranges {0=65536} 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:4: writing smithi18136759-4 from 0 to 65536 tid 1 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:5: write initial oid 5 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:5: seq_num 4 ranges {0=81920} 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:5: writing smithi18136759-5 from 0 to 81920 tid 1 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:6: write initial oid 6 2024-09-17T16:01:33.851 INFO:tasks.rados.rados.0.smithi181.stdout:6: seq_num 5 ranges {0=98304} 2024-09-17T16:01:33.853 INFO:tasks.rados.rados.0.smithi181.stdout:6: writing smithi18136759-6 from 0 to 98304 tid 1 2024-09-17T16:01:33.853 INFO:tasks.rados.rados.0.smithi181.stdout:7: write initial oid 7 2024-09-17T16:01:33.853 INFO:tasks.rados.rados.0.smithi181.stdout:7: seq_num 6 ranges {0=114688} 2024-09-17T16:01:33.853 INFO:tasks.rados.rados.0.smithi181.stdout:7: writing smithi18136759-7 from 0 to 114688 tid 1 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:8: write initial oid 8 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:8: seq_num 7 ranges {0=131072} 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:8: writing smithi18136759-8 from 0 to 131072 tid 1 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:9: write initial oid 9 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:9: seq_num 8 ranges {0=147456} 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:9: writing smithi18136759-9 from 0 to 147456 tid 1 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:10: write initial oid 10 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:10: seq_num 9 ranges {0=163840} 2024-09-17T16:01:33.854 INFO:tasks.rados.rados.0.smithi181.stdout:10: writing smithi18136759-10 from 0 to 163840 tid 1 2024-09-17T16:01:33.855 INFO:tasks.rados.rados.0.smithi181.stdout:11: write initial oid 11 2024-09-17T16:01:33.855 INFO:tasks.rados.rados.0.smithi181.stdout:11: seq_num 10 ranges {0=180224} 2024-09-17T16:01:33.857 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:33.857 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:11: writing smithi18136759-11 from 0 to 180224 tid 1 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:12: write initial oid 12 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:12: seq_num 11 ranges {0=196608} 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:12: writing smithi18136759-12 from 0 to 196608 tid 1 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:13: write initial oid 13 2024-09-17T16:01:33.858 INFO:tasks.rados.rados.0.smithi181.stdout:13: seq_num 12 ranges {0=212992} 2024-09-17T16:01:33.860 INFO:tasks.rados.rados.0.smithi181.stdout:13: writing smithi18136759-13 from 0 to 212992 tid 1 2024-09-17T16:01:33.860 INFO:tasks.rados.rados.0.smithi181.stdout:14: write initial oid 14 2024-09-17T16:01:33.860 INFO:tasks.rados.rados.0.smithi181.stdout:14: seq_num 13 ranges {0=229376} 2024-09-17T16:01:33.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:01:33.862 INFO:tasks.rados.rados.0.smithi181.stdout:14: writing smithi18136759-14 from 0 to 229376 tid 1 2024-09-17T16:01:33.862 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 1 to smithi18136759-1 2024-09-17T16:01:33.862 INFO:tasks.rados.rados.0.smithi181.stdout:15: write initial oid 15 2024-09-17T16:01:33.862 INFO:tasks.rados.rados.0.smithi181.stdout:15: seq_num 14 ranges {0=245760} 2024-09-17T16:01:33.863 INFO:tasks.rados.rados.0.smithi181.stdout:15: writing smithi18136759-15 from 0 to 245760 tid 1 2024-09-17T16:01:33.863 INFO:tasks.rados.rados.0.smithi181.stdout:16: write initial oid 16 2024-09-17T16:01:33.863 INFO:tasks.rados.rados.0.smithi181.stdout:16: seq_num 15 ranges {0=262144} 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:16: writing smithi18136759-16 from 0 to 262144 tid 1 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 1 to smithi18136759-2 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 1 to smithi18136759-3 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 2 to smithi18136759-2 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 1 to smithi18136759-7 2024-09-17T16:01:33.865 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 2 to smithi18136759-3 2024-09-17T16:01:33.866 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 1 to smithi18136759-5 2024-09-17T16:01:33.866 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 2 to smithi18136759-1 2024-09-17T16:01:33.866 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 2 to smithi18136759-5 2024-09-17T16:01:33.867 INFO:tasks.rados.rados.0.smithi181.stdout:5: finishing write tid 3 to smithi18136759-5 2024-09-17T16:01:33.867 INFO:tasks.rados.rados.0.smithi181.stdout:5: oid 5 updating version 0 to 1 2024-09-17T16:01:33.867 INFO:tasks.rados.rados.0.smithi181.stdout:5: oid 5 updating version 1 to 2 2024-09-17T16:01:33.867 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 2 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-17T16:01:33.867 INFO:tasks.rados.rados.0.smithi181.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:1: finishing write tid 3 to smithi18136759-1 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:1: oid 1 updating version 0 to 1 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:1: oid 1 updating version 1 to 2 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 1 to smithi18136759-6 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:1: done (15 left) 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:5: done (14 left) 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:17: write initial oid 17 2024-09-17T16:01:33.868 INFO:tasks.rados.rados.0.smithi181.stdout:17: seq_num 16 ranges {0=278528} 2024-09-17T16:01:33.869 INFO:tasks.rados.rados.0.smithi181.stdout:17: writing smithi18136759-17 from 0 to 278528 tid 1 2024-09-17T16:01:33.869 INFO:tasks.rados.rados.0.smithi181.stdout:18: write initial oid 18 2024-09-17T16:01:33.869 INFO:tasks.rados.rados.0.smithi181.stdout:18: seq_num 17 ranges {0=294912} 2024-09-17T16:01:33.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:18: writing smithi18136759-18 from 0 to 294912 tid 1 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 2 to smithi18136759-7 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:2: finishing write tid 3 to smithi18136759-2 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:2: oid 2 updating version 0 to 1 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:2: oid 2 updating version 1 to 2 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:7: finishing write tid 3 to smithi18136759-7 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:7: oid 7 updating version 0 to 1 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:7: oid 7 updating version 1 to 2 2024-09-17T16:01:33.872 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 1 to smithi18136759-13 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 2 to smithi18136759-13 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:2: done (15 left) 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:7: done (14 left) 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:19: write initial oid 19 2024-09-17T16:01:33.873 INFO:tasks.rados.rados.0.smithi181.stdout:19: seq_num 18 ranges {0=311296} 2024-09-17T16:01:33.874 INFO:tasks.rados.rados.0.smithi181.stdout:19: writing smithi18136759-19 from 0 to 311296 tid 1 2024-09-17T16:01:33.874 INFO:tasks.rados.rados.0.smithi181.stdout:20: write initial oid 20 2024-09-17T16:01:33.874 INFO:tasks.rados.rados.0.smithi181.stdout:20: seq_num 19 ranges {0=327680} 2024-09-17T16:01:33.876 INFO:tasks.rados.rados.0.smithi181.stdout:20: writing smithi18136759-20 from 0 to 327680 tid 1 2024-09-17T16:01:33.876 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.876 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 1 to smithi18136759-12 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 2 to smithi18136759-12 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 1 to smithi18136759-11 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:13: finishing write tid 3 to smithi18136759-13 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:13: oid 13 updating version 0 to 1 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:13: oid 13 updating version 1 to 2 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 2 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 2 to smithi18136759-6 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 2 to smithi18136759-11 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:13: done (15 left) 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:21: write initial oid 21 2024-09-17T16:01:33.877 INFO:tasks.rados.rados.0.smithi181.stdout:21: seq_num 20 ranges {0=344064} 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:21: writing smithi18136759-21 from 0 to 344064 tid 1 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:11: finishing write tid 3 to smithi18136759-11 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:11: oid 11 updating version 0 to 1 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:11: oid 11 updating version 1 to 2 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:12: finishing write tid 3 to smithi18136759-12 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:12: oid 12 updating version 0 to 4 2024-09-17T16:01:33.879 INFO:tasks.rados.rados.0.smithi181.stdout:12: oid 12 version 4 is already newer than 3 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 4 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:6: finishing write tid 3 to smithi18136759-6 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:6: oid 6 updating version 0 to 1 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:6: oid 6 updating version 1 to 2 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 1 to smithi18136759-16 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 1 to smithi18136759-8 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 2 to smithi18136759-8 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 1 to smithi18136759-15 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:6: done (15 left) 2024-09-17T16:01:33.880 INFO:tasks.rados.rados.0.smithi181.stdout:11: done (14 left) 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:12: done (13 left) 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:22: write initial oid 22 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:22: seq_num 21 ranges {0=360448} 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:22: writing smithi18136759-22 from 0 to 360448 tid 1 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:23: write initial oid 23 2024-09-17T16:01:33.881 INFO:tasks.rados.rados.0.smithi181.stdout:23: seq_num 22 ranges {0=376832} 2024-09-17T16:01:33.883 INFO:tasks.rados.rados.0.smithi181.stdout:23: writing smithi18136759-23 from 0 to 376832 tid 1 2024-09-17T16:01:33.883 INFO:tasks.rados.rados.0.smithi181.stdout:24: write initial oid 24 2024-09-17T16:01:33.883 INFO:tasks.rados.rados.0.smithi181.stdout:24: seq_num 23 ranges {0=393216} 2024-09-17T16:01:33.886 INFO:tasks.rados.rados.0.smithi181.stdout:24: writing smithi18136759-24 from 0 to 393216 tid 1 2024-09-17T16:01:33.886 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.886 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 2 to smithi18136759-15 2024-09-17T16:01:33.886 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 1 to smithi18136759-9 2024-09-17T16:01:33.886 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 2 to smithi18136759-9 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 1 to smithi18136759-17 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 1 to smithi18136759-4 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:15: finishing write tid 3 to smithi18136759-15 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:15: oid 15 updating version 0 to 4 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:15: oid 15 version 4 is already newer than 3 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 2 to smithi18136759-4 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:3: finishing write tid 3 to smithi18136759-3 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:3: oid 3 updating version 0 to 1 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:3: oid 3 updating version 1 to 2 2024-09-17T16:01:33.887 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:9: finishing write tid 3 to smithi18136759-9 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:9: oid 9 updating version 0 to 3 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:9: oid 9 updating version 3 to 4 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 4 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 1 to smithi18136759-20 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:4: finishing write tid 3 to smithi18136759-4 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:4: oid 4 updating version 0 to 3 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:4: oid 4 updating version 3 to 4 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 4 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:3: done (15 left) 2024-09-17T16:01:33.888 INFO:tasks.rados.rados.0.smithi181.stdout:4: done (14 left) 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:9: done (13 left) 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:15: done (12 left) 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:25: write initial oid 25 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:25: seq_num 24 ranges {0=409600} 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:25: writing smithi18136759-25 from 0 to 409600 tid 1 2024-09-17T16:01:33.889 INFO:tasks.rados.rados.0.smithi181.stdout:26: write initial oid 26 2024-09-17T16:01:33.890 INFO:tasks.rados.rados.0.smithi181.stdout:26: seq_num 25 ranges {0=425984} 2024-09-17T16:01:33.892 INFO:tasks.rados.rados.0.smithi181.stdout:26: writing smithi18136759-26 from 0 to 425984 tid 1 2024-09-17T16:01:33.892 INFO:tasks.rados.rados.0.smithi181.stdout:27: write initial oid 27 2024-09-17T16:01:33.892 INFO:tasks.rados.rados.0.smithi181.stdout:27: seq_num 26 ranges {0=442368} 2024-09-17T16:01:33.893 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:33.894 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:33.895 INFO:tasks.rados.rados.0.smithi181.stdout:27: writing smithi18136759-27 from 0 to 442368 tid 1 2024-09-17T16:01:33.895 INFO:tasks.rados.rados.0.smithi181.stdout:28: write initial oid 28 2024-09-17T16:01:33.895 INFO:tasks.rados.rados.0.smithi181.stdout:28: seq_num 27 ranges {0=458752} 2024-09-17T16:01:33.899 INFO:tasks.rados.rados.0.smithi181.stdout:28: writing smithi18136759-28 from 0 to 458752 tid 1 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 2 to smithi18136759-20 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 2 to smithi18136759-17 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 1 to smithi18136759-19 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:20: finishing write tid 3 to smithi18136759-20 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:20: oid 20 updating version 0 to 3 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:20: oid 20 updating version 3 to 4 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 4 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-17T16:01:33.900 INFO:tasks.rados.rados.0.smithi181.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 2 to smithi18136759-16 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 2 to smithi18136759-19 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 1 to smithi18136759-23 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 2 to smithi18136759-23 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:16: finishing write tid 3 to smithi18136759-16 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:16: oid 16 updating version 0 to 1 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:16: oid 16 updating version 1 to 2 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:17: finishing write tid 3 to smithi18136759-17 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:17: oid 17 updating version 0 to 2 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:17: oid 17 version 2 is already newer than 1 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:01:33.901 INFO:tasks.rados.rados.0.smithi181.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 1 to smithi18136759-22 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 2 to smithi18136759-22 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:19: finishing write tid 3 to smithi18136759-19 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:19: oid 19 updating version 0 to 2 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:19: oid 19 version 2 is already newer than 1 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 1 to smithi18136759-21 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 2 to smithi18136759-21 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: finishing write tid 3 to smithi18136759-22 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: oid 22 updating version 0 to 4 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: oid 22 version 4 is already newer than 3 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-17T16:01:33.902 INFO:tasks.rados.rados.0.smithi181.stdout:23: finishing write tid 3 to smithi18136759-23 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:23: oid 23 updating version 0 to 6 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:23: oid 23 version 6 is already newer than 5 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:21: finishing write tid 3 to smithi18136759-21 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:21: oid 21 updating version 0 to 1 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:21: oid 21 updating version 1 to 2 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 2 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 1 to smithi18136759-25 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:16: done (15 left) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:17: done (14 left) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:19: done (13 left) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:20: done (12 left) 2024-09-17T16:01:33.903 INFO:tasks.rados.rados.0.smithi181.stdout:21: done (11 left) 2024-09-17T16:01:33.904 INFO:tasks.rados.rados.0.smithi181.stdout:22: done (10 left) 2024-09-17T16:01:33.904 INFO:tasks.rados.rados.0.smithi181.stdout:23: done (9 left) 2024-09-17T16:01:33.904 INFO:tasks.rados.rados.0.smithi181.stdout:29: write initial oid 29 2024-09-17T16:01:33.904 INFO:tasks.rados.rados.0.smithi181.stdout:29: seq_num 28 ranges {0=475136} 2024-09-17T16:01:33.904 INFO:tasks.rados.rados.0.smithi181.stdout:29: writing smithi18136759-29 from 0 to 475136 tid 1 2024-09-17T16:01:33.905 INFO:tasks.rados.rados.0.smithi181.stdout:30: write initial oid 30 2024-09-17T16:01:33.905 INFO:tasks.rados.rados.0.smithi181.stdout:30: seq_num 29 ranges {0=491520} 2024-09-17T16:01:33.905 INFO:tasks.rados.rados.0.smithi181.stdout:30: writing smithi18136759-30 from 0 to 491520 tid 1 2024-09-17T16:01:33.905 INFO:tasks.rados.rados.0.smithi181.stdout:31: write initial oid 31 2024-09-17T16:01:33.905 INFO:tasks.rados.rados.0.smithi181.stdout:31: seq_num 30 ranges {0=507904} 2024-09-17T16:01:33.907 INFO:tasks.rados.rados.0.smithi181.stdout:31: writing smithi18136759-31 from 0 to 507904 tid 1 2024-09-17T16:01:33.907 INFO:tasks.rados.rados.0.smithi181.stdout:32: write initial oid 32 2024-09-17T16:01:33.907 INFO:tasks.rados.rados.0.smithi181.stdout:32: seq_num 31 ranges {0=524288} 2024-09-17T16:01:33.910 INFO:tasks.rados.rados.0.smithi181.stdout:32: writing smithi18136759-32 from 0 to 524288 tid 1 2024-09-17T16:01:33.910 INFO:tasks.rados.rados.0.smithi181.stdout:33: write initial oid 33 2024-09-17T16:01:33.910 INFO:tasks.rados.rados.0.smithi181.stdout:33: seq_num 32 ranges {0=540672} 2024-09-17T16:01:33.914 INFO:tasks.rados.rados.0.smithi181.stdout:33: writing smithi18136759-33 from 0 to 540672 tid 1 2024-09-17T16:01:33.914 INFO:tasks.rados.rados.0.smithi181.stdout:34: write initial oid 34 2024-09-17T16:01:33.914 INFO:tasks.rados.rados.0.smithi181.stdout:34: seq_num 33 ranges {0=557056} 2024-09-17T16:01:33.917 INFO:tasks.rados.rados.0.smithi181.stdout:34: writing smithi18136759-34 from 0 to 557056 tid 1 2024-09-17T16:01:33.917 INFO:tasks.rados.rados.0.smithi181.stdout:35: write initial oid 35 2024-09-17T16:01:33.917 INFO:tasks.rados.rados.0.smithi181.stdout:35: seq_num 34 ranges {0=573440} 2024-09-17T16:01:33.922 INFO:tasks.rados.rados.0.smithi181.stdout:35: writing smithi18136759-35 from 0 to 573440 tid 1 2024-09-17T16:01:33.922 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 1 to smithi18136759-10 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 2 to smithi18136759-10 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 1 to smithi18136759-14 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 2 to smithi18136759-14 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 1 to smithi18136759-18 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 2 to smithi18136759-18 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 1 to smithi18136759-28 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 1 to smithi18136759-27 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:8: finishing write tid 3 to smithi18136759-8 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:8: oid 8 updating version 0 to 1 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:8: oid 8 updating version 1 to 2 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 2 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-17T16:01:33.923 INFO:tasks.rados.rados.0.smithi181.stdout:10: finishing write tid 3 to smithi18136759-10 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:10: oid 10 updating version 0 to 4 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:10: oid 10 version 4 is already newer than 3 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:14: finishing write tid 3 to smithi18136759-14 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:14: oid 14 updating version 0 to 5 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:14: oid 14 updating version 5 to 6 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 2 to smithi18136759-28 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:18: finishing write tid 3 to smithi18136759-18 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:18: oid 18 updating version 0 to 8 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:18: oid 18 version 8 is already newer than 7 2024-09-17T16:01:33.924 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 8 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 2 to smithi18136759-25 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:25: finishing write tid 3 to smithi18136759-25 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:25: oid 25 updating version 0 to 4 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:25: oid 25 version 4 is already newer than 3 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 25 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:28: finishing write tid 3 to smithi18136759-28 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:28: oid 28 updating version 0 to 6 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:28: oid 28 version 6 is already newer than 5 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-17T16:01:33.925 INFO:tasks.rados.rados.0.smithi181.stdout:8: done (15 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:10: done (14 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:14: done (13 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:18: done (12 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:25: done (11 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:28: done (10 left) 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:36: write initial oid 36 2024-09-17T16:01:33.926 INFO:tasks.rados.rados.0.smithi181.stdout:36: seq_num 35 ranges {0=589824} 2024-09-17T16:01:33.927 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:33.927 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:33.927 INFO:tasks.rados.rados.0.smithi181.stdout:36: writing smithi18136759-36 from 0 to 589824 tid 1 2024-09-17T16:01:33.927 INFO:tasks.rados.rados.0.smithi181.stdout:37: write initial oid 37 2024-09-17T16:01:33.927 INFO:tasks.rados.rados.0.smithi181.stdout:37: seq_num 36 ranges {0=606208} 2024-09-17T16:01:33.928 INFO:tasks.rados.rados.0.smithi181.stdout:37: writing smithi18136759-37 from 0 to 606208 tid 1 2024-09-17T16:01:33.928 INFO:tasks.rados.rados.0.smithi181.stdout:38: write initial oid 38 2024-09-17T16:01:33.928 INFO:tasks.rados.rados.0.smithi181.stdout:38: seq_num 37 ranges {0=622592} 2024-09-17T16:01:33.931 INFO:tasks.rados.rados.0.smithi181.stdout:38: writing smithi18136759-38 from 0 to 622592 tid 1 2024-09-17T16:01:33.932 INFO:tasks.rados.rados.0.smithi181.stdout:39: write initial oid 39 2024-09-17T16:01:33.932 INFO:tasks.rados.rados.0.smithi181.stdout:39: seq_num 38 ranges {0=638976} 2024-09-17T16:01:33.935 INFO:tasks.rados.rados.0.smithi181.stdout:39: writing smithi18136759-39 from 0 to 638976 tid 1 2024-09-17T16:01:33.935 INFO:tasks.rados.rados.0.smithi181.stdout:40: write initial oid 40 2024-09-17T16:01:33.935 INFO:tasks.rados.rados.0.smithi181.stdout:40: seq_num 39 ranges {0=671744} 2024-09-17T16:01:33.937 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:33.934+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:33.939 INFO:tasks.rados.rados.0.smithi181.stdout:40: writing smithi18136759-40 from 0 to 671744 tid 1 2024-09-17T16:01:33.939 INFO:tasks.rados.rados.0.smithi181.stdout:41: write initial oid 41 2024-09-17T16:01:33.939 INFO:tasks.rados.rados.0.smithi181.stdout:41: seq_num 40 ranges {0=688128} 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:41: writing smithi18136759-41 from 0 to 688128 tid 1 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 2 to smithi18136759-27 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:27: finishing write tid 3 to smithi18136759-27 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:27: oid 27 updating version 0 to 7 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:27: oid 27 updating version 7 to 8 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-17T16:01:33.944 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 1 to smithi18136759-31 2024-09-17T16:01:33.945 INFO:tasks.rados.rados.0.smithi181.stdout:27: done (15 left) 2024-09-17T16:01:33.945 INFO:tasks.rados.rados.0.smithi181.stdout:42: write initial oid 42 2024-09-17T16:01:33.945 INFO:tasks.rados.rados.0.smithi181.stdout:42: seq_num 41 ranges {0=704512} 2024-09-17T16:01:33.949 INFO:tasks.rados.rados.0.smithi181.stdout:42: writing smithi18136759-42 from 0 to 704512 tid 1 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 1 to smithi18136759-30 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 1 to smithi18136759-24 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 2 to smithi18136759-24 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: finishing write tid 3 to smithi18136759-24 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: oid 24 updating version 0 to 10 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: oid 24 version 10 is already newer than 9 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 10 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 1 to smithi18136759-29 2024-09-17T16:01:33.950 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 2 to smithi18136759-29 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 2 to smithi18136759-31 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:31: finishing write tid 3 to smithi18136759-31 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:31: oid 31 updating version 0 to 5 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:31: oid 31 updating version 5 to 6 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:29: finishing write tid 3 to smithi18136759-29 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:29: oid 29 updating version 0 to 11 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:29: oid 29 updating version 11 to 12 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 1 to smithi18136759-33 2024-09-17T16:01:33.951 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 1 to smithi18136759-32 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:24: done (15 left) 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:29: done (14 left) 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:31: done (13 left) 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:43: write initial oid 43 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:43: seq_num 42 ranges {0=720896} 2024-09-17T16:01:33.952 INFO:tasks.rados.rados.0.smithi181.stdout:43: writing smithi18136759-43 from 0 to 720896 tid 1 2024-09-17T16:01:33.953 INFO:tasks.rados.rados.0.smithi181.stdout:44: write initial oid 44 2024-09-17T16:01:33.953 INFO:tasks.rados.rados.0.smithi181.stdout:44: seq_num 43 ranges {0=737280} 2024-09-17T16:01:33.954 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:33.954 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:33.957 INFO:tasks.rados.rados.0.smithi181.stdout:44: writing smithi18136759-44 from 0 to 737280 tid 1 2024-09-17T16:01:33.957 INFO:tasks.rados.rados.0.smithi181.stdout:45: write initial oid 45 2024-09-17T16:01:33.957 INFO:tasks.rados.rados.0.smithi181.stdout:45: seq_num 44 ranges {0=753664} 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout:45: writing smithi18136759-45 from 0 to 753664 tid 1 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 2 to smithi18136759-33 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 1 to smithi18136759-34 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 2 to smithi18136759-34 2024-09-17T16:01:33.962 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 1 to smithi18136759-35 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:33: finishing write tid 3 to smithi18136759-33 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:33: oid 33 updating version 0 to 3 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:33: oid 33 updating version 3 to 4 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 4 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 2 to smithi18136759-35 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:34: finishing write tid 3 to smithi18136759-34 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:34: oid 34 updating version 0 to 3 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:34: oid 34 updating version 3 to 4 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 4 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 1 to smithi18136759-36 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 2 to smithi18136759-36 2024-09-17T16:01:33.963 INFO:tasks.rados.rados.0.smithi181.stdout:36: finishing write tid 3 to smithi18136759-36 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:36: oid 36 updating version 0 to 4 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:36: oid 36 version 4 is already newer than 3 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 4 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 1 to smithi18136759-37 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:33: done (15 left) 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:34: done (14 left) 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:36: done (13 left) 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:46: write initial oid 46 2024-09-17T16:01:33.964 INFO:tasks.rados.rados.0.smithi181.stdout:46: seq_num 45 ranges {0=770048} 2024-09-17T16:01:33.966 INFO:tasks.rados.rados.0.smithi181.stdout:46: writing smithi18136759-46 from 0 to 770048 tid 1 2024-09-17T16:01:33.966 INFO:tasks.rados.rados.0.smithi181.stdout:47: write initial oid 47 2024-09-17T16:01:33.966 INFO:tasks.rados.rados.0.smithi181.stdout:47: seq_num 46 ranges {0=786432} 2024-09-17T16:01:33.971 INFO:tasks.rados.rados.0.smithi181.stdout:47: writing smithi18136759-47 from 0 to 786432 tid 1 2024-09-17T16:01:33.971 INFO:tasks.rados.rados.0.smithi181.stdout:48: write initial oid 48 2024-09-17T16:01:33.971 INFO:tasks.rados.rados.0.smithi181.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-17T16:01:33.973 INFO:tasks.rados.rados.0.smithi181.stdout:48: writing smithi18136759-48 from 0 to 425984 tid 1 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:48: writing smithi18136759-48 from 425984 to 802816 tid 2 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 2 to smithi18136759-37 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 1 to smithi18136759-38 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:37: finishing write tid 3 to smithi18136759-37 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:37: oid 37 updating version 0 to 2 2024-09-17T16:01:33.976 INFO:tasks.rados.rados.0.smithi181.stdout:37: oid 37 version 2 is already newer than 1 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 2 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 2 to smithi18136759-38 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 1 to smithi18136759-40 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 2 to smithi18136759-40 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:37: done (15 left) 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:49: write initial oid 49 2024-09-17T16:01:33.977 INFO:tasks.rados.rados.0.smithi181.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-17T16:01:33.978 INFO:tasks.rados.rados.0.smithi181.stdout:49: writing smithi18136759-49 from 0 to 442368 tid 1 2024-09-17T16:01:33.981 INFO:tasks.rados.rados.0.smithi181.stdout:49: writing smithi18136759-49 from 442368 to 819200 tid 2 2024-09-17T16:01:33.981 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:38: finishing write tid 3 to smithi18136759-38 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:38: oid 38 updating version 0 to 4 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:38: oid 38 version 4 is already newer than 3 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 4 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:40: finishing write tid 3 to smithi18136759-40 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:40: oid 40 updating version 0 to 2 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:40: oid 40 version 2 is already newer than 1 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 2 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:38: done (15 left) 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:40: done (14 left) 2024-09-17T16:01:33.982 INFO:tasks.rados.rados.0.smithi181.stdout:50: write initial oid 50 2024-09-17T16:01:33.983 INFO:tasks.rados.rados.0.smithi181.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-17T16:01:33.983 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:33.983 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:33.984 INFO:tasks.rados.rados.0.smithi181.stdout:50: writing smithi18136759-50 from 0 to 458752 tid 1 2024-09-17T16:01:33.989 INFO:tasks.rados.rados.0.smithi181.stdout:50: writing smithi18136759-50 from 458752 to 835584 tid 2 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:51: read oid 16 snap -1 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:51: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:35: finishing write tid 3 to smithi18136759-35 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:35: oid 35 updating version 0 to 6 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:35: oid 35 version 6 is already newer than 5 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 2 to smithi18136759-32 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 1 to smithi18136759-42 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:35: done (15 left) 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:52: read oid 15 snap -1 2024-09-17T16:01:33.990 INFO:tasks.rados.rados.0.smithi181.stdout:52: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 16 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:32: finishing write tid 3 to smithi18136759-32 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:32: oid 32 updating version 0 to 4 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:32: oid 32 version 4 is already newer than 3 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 1 to smithi18136759-41 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:32: done (15 left) 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:53: rollback oid 8 current snap is 0 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:53: done (15 left) 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:54: delete oid 1 current snap is 0 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 2 to smithi18136759-41 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 1 to smithi18136759-43 2024-09-17T16:01:33.991 INFO:tasks.rados.rados.0.smithi181.stdout:41: finishing write tid 3 to smithi18136759-41 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:41: oid 41 updating version 0 to 5 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:41: oid 41 updating version 5 to 6 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 2 to smithi18136759-43 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 1 to smithi18136759-44 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 2 to smithi18136759-44 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: finishing write tid 3 to smithi18136759-44 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: oid 44 updating version 0 to 3 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: oid 44 updating version 3 to 4 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 4 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-17T16:01:33.992 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 1 to smithi18136759-45 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 2 to smithi18136759-45 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:45: finishing write tid 3 to smithi18136759-45 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:45: oid 45 updating version 0 to 6 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:45: oid 45 version 6 is already newer than 5 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 1 to smithi18136759-46 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 2 to smithi18136759-46 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: finishing write tid 3 to smithi18136759-46 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: oid 46 updating version 0 to 6 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: oid 46 version 6 is already newer than 5 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-17T16:01:33.993 INFO:tasks.rados.rados.0.smithi181.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 1 to smithi18136759-47 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 1 to smithi18136759-48 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:43: finishing write tid 3 to smithi18136759-43 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:43: oid 43 updating version 0 to 6 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:43: oid 43 version 6 is already newer than 5 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 2 to smithi18136759-47 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 2 to smithi18136759-30 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 2 to smithi18136759-48 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 3 to smithi18136759-48 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:47: finishing write tid 3 to smithi18136759-47 2024-09-17T16:01:33.994 INFO:tasks.rados.rados.0.smithi181.stdout:47: oid 47 updating version 0 to 3 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:47: oid 47 updating version 3 to 4 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 4 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 2 to smithi18136759-42 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:48: finishing write tid 4 to smithi18136759-48 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 updating version 0 to 5 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 updating version 5 to 6 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:48: oid 48 updating version 6 to 7 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 1 to smithi18136759-49 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 2 to smithi18136759-49 2024-09-17T16:01:33.995 INFO:tasks.rados.rados.0.smithi181.stdout:42: finishing write tid 3 to smithi18136759-42 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:42: oid 42 updating version 0 to 6 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:42: oid 42 version 6 is already newer than 5 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:41: done (15 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:42: done (14 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:43: done (13 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:44: done (12 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:45: done (11 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:46: done (10 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:47: done (9 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:48: done (8 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:54: done (7 left) 2024-09-17T16:01:33.996 INFO:tasks.rados.rados.0.smithi181.stdout:55: delete oid 18 current snap is 0 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 1 to smithi18136759-50 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:30: finishing write tid 3 to smithi18136759-30 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:30: oid 30 updating version 0 to 6 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:30: oid 30 version 6 is already newer than 5 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-17T16:01:33.997 INFO:tasks.rados.rados.0.smithi181.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-17T16:01:33.997 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:33.997 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:33.997 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:01:33.998 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:01:33.998 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 2 to smithi18136759-50 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 3 to smithi18136759-50 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:52: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 120514 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 196494 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 50685 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:50: finishing write tid 4 to smithi18136759-50 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 updating version 0 to 8 2024-09-17T16:01:33.999 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 updating version 8 to 10 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:50: oid 50 version 10 is already newer than 9 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 10 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:30: done (7 left) 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:50: done (6 left) 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:52: done (5 left) 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:55: done (4 left) 2024-09-17T16:01:34.000 INFO:tasks.rados.rados.0.smithi181.stdout:56: snap_create 2024-09-17T16:01:34.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:01:34.008 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:34.031 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:34.031 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:34.038 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:34.036+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:34.053 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:34.054 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:34.105 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 1 to smithi18136759-26 2024-09-17T16:01:34.107 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 2 to smithi18136759-26 2024-09-17T16:01:34.107 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 1 to smithi18136759-39 2024-09-17T16:01:34.108 INFO:tasks.rados.rados.0.smithi181.stdout:26: finishing write tid 3 to smithi18136759-26 2024-09-17T16:01:34.108 INFO:tasks.rados.rados.0.smithi181.stdout:26: oid 26 updating version 0 to 1 2024-09-17T16:01:34.108 INFO:tasks.rados.rados.0.smithi181.stdout:26: oid 26 updating version 1 to 2 2024-09-17T16:01:34.108 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 2 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T16:01:34.108 INFO:tasks.rados.rados.0.smithi181.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-17T16:01:34.110 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 2 to smithi18136759-39 2024-09-17T16:01:34.111 INFO:tasks.rados.rados.0.smithi181.stdout:39: finishing write tid 3 to smithi18136759-39 2024-09-17T16:01:34.111 INFO:tasks.rados.rados.0.smithi181.stdout:39: oid 39 updating version 0 to 3 2024-09-17T16:01:34.111 INFO:tasks.rados.rados.0.smithi181.stdout:39: oid 39 updating version 3 to 4 2024-09-17T16:01:34.111 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-17T16:01:34.111 INFO:tasks.rados.rados.0.smithi181.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-17T16:01:34.138 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:34.135+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:34.149 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:01:34.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.286397189, 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053801109999999999, 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860418+0000", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020234499999999999 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860448+0000", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0692999999999999e-05 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860458+0000", 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6460000000000005e-06 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860528+0000", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0272999999999996e-05 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860536+0000", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1040000000000007e-06 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865550+0000", 2024-09-17T16:01:34.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050133510000000001 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865595+0000", 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5396999999999999e-05 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:01:34.154 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.28523554899999998, 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042468009999999997, 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861378+0000", 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999999e-07 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861403+0000", 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5069999999999999e-05 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861434+0000", 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1396000000000002e-05 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861442+0000", 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4500000000000004e-06 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861497+0000", 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4997000000000002e-05 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861508+0000", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0360000000000001e-05 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865601+0000", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040936439999999996 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865624+0000", 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2101999999999999e-05 2024-09-17T16:01:34.158 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.28236146499999998, 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051579750000000004, 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:01:34.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864260+0000", 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5769999999999994e-06 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864296+0000", 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6227999999999999e-05 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864304+0000", 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0090000000000007e-06 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864392+0000", 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8549999999999998e-05 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864402+0000", 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3279999999999995e-06 2024-09-17T16:01:34.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869377+0000", 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004975045 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869409+0000", 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1953999999999999e-05 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.163 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=119, reqid=client.4560.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.864765+0000", 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.281847344, 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003855673, 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864765+0000", 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864765+0000", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864765+0000", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7399999999999999e-07 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864773+0000", 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7209999999999994e-06 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864824+0000", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0630000000000001e-05 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864832+0000", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3829999999999992e-06 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864893+0000", 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0943999999999998e-05 2024-09-17T16:01:34.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864904+0000", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0854e-05 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868580+0000", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003675878 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868621+0000", 2024-09-17T16:01:34.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0988999999999997e-05 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.27914905000000001, 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.168 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.27875933800000002, 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=7, reqid=client.4560.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.868286+0000", 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.27832586399999998, 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028250139999999998, 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868286+0000", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868286+0000", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868287+0000", 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0599999999999999e-07 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868298+0000", 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0545e-05 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868326+0000", 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7832999999999999e-05 2024-09-17T16:01:34.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868334+0000", 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5650000000000002e-06 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868403+0000", 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8850000000000007e-05 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868411+0000", 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7270000000000007e-06 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871076+0000", 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026651819999999999 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871111+0000", 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5506000000000002e-05 2024-09-17T16:01:34.181 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.269338929, 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:01:34.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=12, reqid=client.4560.0:64, at_version=35'3, trim_to=0'0, roll_forward_to=35'1))", 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.881733+0000", 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.264878907, 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036100469999999999, 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881733+0000", 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881733+0000", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881734+0000", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0240000000000001e-06 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881899+0000", 2024-09-17T16:01:34.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016469600000000001 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881933+0000", 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3408999999999998e-05 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881942+0000", 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.516e-06 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882006+0000", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3528999999999994e-05 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882017+0000", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.152e-05 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885304+0000", 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032868089999999999 2024-09-17T16:01:34.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885343+0000", 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9544e-05 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=13, reqid=client.4560.0:65, at_version=35'4, trim_to=0'0, roll_forward_to=35'1))", 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.882458+0000", 2024-09-17T16:01:34.191 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.26415450499999998, 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0029133029999999999, 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882458+0000", 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882458+0000", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882458+0000", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882475+0000", 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6084999999999999e-05 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882504+0000", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9085e-05 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882515+0000", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124000000000001e-05 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882574+0000", 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9669000000000001e-05 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.882583+0000", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8939999999999999e-06 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885351+0000", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027680840000000001 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885371+0000", 2024-09-17T16:01:34.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9698e-05 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=22, reqid=client.4560.0:89, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.905863+0000", 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.240749608, 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028058800000000002, 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905863+0000", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905863+0000", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905864+0000", 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:01:34.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905884+0000", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0482000000000001e-05 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905914+0000", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9731e-05 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905923+0000", 2024-09-17T16:01:34.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7360000000000007e-06 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905986+0000", 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3287000000000006e-05 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905995+0000", 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1619999999999997e-06 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908624+0000", 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002629115 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908669+0000", 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4428999999999998e-05 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.200 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21465340399999999, 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:01:34.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:01:34.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21427100499999999, 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.206 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:01:34.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.211 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21423521200000001, 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:01:34.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:01:34.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21420704700000001, 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.217 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:01:34.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.21393352800000001, 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:01:34.222 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:01:34.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:01:34.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:01:34.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.213922472, 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:01:34.228 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:01:34.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.18116731899999999, 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.233 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:01:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:01:34.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.180678269, 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.239 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:01:34.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:01:34.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:01:34.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:01:34.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.246 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.18063741899999999, 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:01:34.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:01:34.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.251 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:34.252 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:34.252 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:34.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:01:34.254 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:34.251+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:34.354 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:34.352+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:34.361 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:34.362 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:34.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:01:34.414 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:34.414 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:34.437 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:34.437 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:34.455 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:34.453+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:34.520 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:34.520 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:34.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:01:34.572 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:34.570+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:34.609 INFO:tasks.rados.rados.0.smithi181.stdout:51: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:01:34.609 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:34.611 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:34.614 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:34.635 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:34.636 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:34.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:01:34.686 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:34.684+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:34.697 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 3 to smithi18136759-49 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:49: finishing write tid 4 to smithi18136759-49 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 updating version 0 to 9 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 version 9 is already newer than 8 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:49: oid 49 version 9 is already newer than 7 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T16:01:34.698 INFO:tasks.rados.rados.0.smithi181.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-17T16:01:34.758 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:34.756+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:34.793 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:34.793 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:34.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:01:34.854 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:01:34.854 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:01:34.857 INFO:tasks.rados.rados.0.smithi181.stdout:26: done (4 left) 2024-09-17T16:01:34.857 INFO:tasks.rados.rados.0.smithi181.stdout:39: done (3 left) 2024-09-17T16:01:34.857 INFO:tasks.rados.rados.0.smithi181.stdout:49: done (2 left) 2024-09-17T16:01:34.857 INFO:tasks.rados.rados.0.smithi181.stdout:51: done (1 left) 2024-09-17T16:01:34.857 INFO:tasks.rados.rados.0.smithi181.stdout:56: done (0 left) 2024-09-17T16:01:34.858 INFO:tasks.rados.rados.0.smithi181.stdout:57: read oid 12 snap -1 2024-09-17T16:01:34.858 INFO:tasks.rados.rados.0.smithi181.stdout:57: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T16:01:34.858 INFO:tasks.rados.rados.0.smithi181.stdout:58: snap_create 2024-09-17T16:01:34.859 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:34.857+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:34.868 INFO:tasks.rados.rados.0.smithi181.stdout:57: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T16:01:34.868 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 24230 2024-09-17T16:01:34.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-17T16:01:34.871 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:34.872 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:34.912 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:34.913 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:34.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:01:34.960 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:34.958+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:35.060 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:35.058+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:35.072 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.072 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:01:35.161 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:35.160+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:35.188 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.189 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:01:35.282 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:35.280+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:35.345 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.345 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:01:35.363 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:35.361+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:35.464 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:35.461+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:35.501 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.501 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:01:35.565 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:35.562+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:35.617 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.617 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:01:35.667 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:35.665+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:35.767 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:35.765+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:35.773 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.773 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:01:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:01:35.861 INFO:tasks.rados.rados.0.smithi181.stdout:57: done (1 left) 2024-09-17T16:01:35.861 INFO:tasks.rados.rados.0.smithi181.stdout:58: done (0 left) 2024-09-17T16:01:35.861 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 40 current snap is 2 2024-09-17T16:01:35.861 INFO:tasks.rados.rados.0.smithi181.stdout:59: seq_num 52 ranges {671744=507904,1179648=376832} 2024-09-17T16:01:35.867 INFO:tasks.rados.rados.0.smithi181.stdout:59: writing smithi18136759-40 from 671744 to 1179648 tid 1 2024-09-17T16:01:35.867 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:35.866+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:35.869 INFO:tasks.rados.rados.0.smithi181.stdout:59: writing smithi18136759-40 from 1179648 to 1556480 tid 2 2024-09-17T16:01:35.869 INFO:tasks.rados.rados.0.smithi181.stdout:60: snap_create 2024-09-17T16:01:35.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-17T16:01:35.877 INFO:tasks.rados.rados.0.smithi181.stdout:59: finishing write tid 1 to smithi18136759-40 2024-09-17T16:01:35.877 INFO:tasks.rados.rados.0.smithi181.stdout:59: finishing write tid 2 to smithi18136759-40 2024-09-17T16:01:35.887 INFO:tasks.rados.rados.0.smithi181.stdout:59: finishing write tid 3 to smithi18136759-40 2024-09-17T16:01:35.888 INFO:tasks.rados.rados.0.smithi181.stdout:59: finishing write tid 4 to smithi18136759-40 2024-09-17T16:01:35.889 INFO:tasks.rados.rados.0.smithi181.stdout:59: oid 40 updating version 0 to 7 2024-09-17T16:01:35.889 INFO:tasks.rados.rados.0.smithi181.stdout:59: oid 40 updating version 7 to 8 2024-09-17T16:01:35.889 INFO:tasks.rados.rados.0.smithi181.stdout:59: oid 40 version 8 is already newer than 6 2024-09-17T16:01:35.889 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 8 (ObjNum 52 snap 2 seq_num 52) dirty exists 2024-09-17T16:01:35.889 INFO:tasks.rados.rados.0.smithi181.stdout:59: left oid 40 (ObjNum 52 snap 2 seq_num 52) 2024-09-17T16:01:35.894 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:35.894 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:35.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:01:35.968 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:35.965+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:36.025 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.025 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:01:36.069 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:36.067+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:36.171 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:36.168+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:36.182 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.182 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:01:36.272 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:36.270+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:36.338 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.338 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:01:36.373 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:36.370+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:36.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:01:36.455 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.455 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.463 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:01:36.474 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:36.472+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:36.570 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.571 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.575 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:36.572+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:36.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:01:36.584 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:36.585 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:36.606 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:36.607 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:36.628 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:36.629 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:36.650 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:36.651 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:36.671 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:36.671 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:36.675 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:36.674+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:36.686 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:36.686 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:36.692 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:36.692 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:36.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:01:36.713 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:36.714 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:36.735 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:36.735 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:36.755 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:36.755 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:36.777 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:36.777 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:36.777 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:36.775+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:36.799 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:36.799 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:36.802 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:36.802 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:36.802 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:36.802 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:36.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:01:36.821 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:36.821 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:36.841 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:36.841 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:01:36.862 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:36.863 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:36.867 INFO:tasks.rados.rados.0.smithi181.stdout:59: done (1 left) 2024-09-17T16:01:36.867 INFO:tasks.rados.rados.0.smithi181.stdout:60: done (0 left) 2024-09-17T16:01:36.867 INFO:tasks.rados.rados.0.smithi181.stdout:61: copy_from oid 38 from oid 27 current snap is 3 2024-09-17T16:01:36.868 INFO:tasks.rados.rados.0.smithi181.stdout:62: delete oid 45 current snap is 3 2024-09-17T16:01:36.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:01:36.875 INFO:tasks.rados.rados.0.smithi181.stdout:62: done (1 left) 2024-09-17T16:01:36.875 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 20 current snap is 3 2024-09-17T16:01:36.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: seq_num 54 ranges {327680=540672,868352=376832} 2024-09-17T16:01:36.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-17T16:01:36.880 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:36.877+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:36.880 INFO:tasks.rados.rados.0.smithi181.stdout:63: writing smithi18136759-20 from 327680 to 868352 tid 1 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:63: writing smithi18136759-20 from 868352 to 1245184 tid 2 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:64: read oid 3 snap -1 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:64: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:61: finishing copy_from racing read to smithi18136759-38 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:61: finishing copy_from to smithi18136759-38 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:61: done (2 left) 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:65: rollback oid 45 current snap is 3 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 45 to 0 2024-09-17T16:01:36.883 INFO:tasks.rados.rados.0.smithi181.stdout:66: snap_remove snap 2 2024-09-17T16:01:36.892 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:36.890+0000 7f8627fff640 0 -- 172.21.15.6:0/179977771 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7f86282212a0 msgr2=0x7f8628223690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:36.901 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:36.919 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:36.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:01:36.928 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:36.978 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:36.976+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:37.036 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4296.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.265011967, 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012209969999999999, 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-07 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768818+0000", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000006e-06 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768864+0000", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5797999999999998e-05 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768875+0000", 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1610000000000001e-05 2024-09-17T16:01:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768958+0000", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3187000000000002e-05 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768978+0000", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0001e-05 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.769987+0000", 2024-09-17T16:01:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001008793 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.770029+0000", 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1938999999999999e-05 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4296.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.266334903000001, 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001137285, 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767486+0000", 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767494+0000", 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0830000000000004e-06 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767563+0000", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9461999999999993e-05 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767579+0000", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5659999999999999e-05 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767700+0000", 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012102700000000001 2024-09-17T16:01:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767745+0000", 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4975000000000002e-05 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.768579+0000", 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083372899999999996 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.768622+0000", 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3769000000000001e-05 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:7, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1798755550000002, 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001460104, 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853946+0000", 2024-09-17T16:01:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7230000000000001e-06 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854002+0000", 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677999999999998e-05 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854062+0000", 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-05 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854075+0000", 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2747e-05 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854189+0000", 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114376 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854203+0000", 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4100000000000001e-05 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855346+0000", 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011423010000000001 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855405+0000", 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8949000000000001e-05 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=1, reqid=client.4560.0:8, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1780145910000002, 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011212939999999999, 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855806+0000", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5999999999999991e-07 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855829+0000", 2024-09-17T16:01:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2616000000000002e-05 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855889+0000", 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0028000000000002e-05 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855901+0000", 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1732999999999999e-05 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855989+0000", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8177999999999995e-05 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855999+0000", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6870000000000006e-06 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856888+0000", 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088945699999999996 2024-09-17T16:01:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856927+0000", 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8634999999999999e-05 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1774929169999999, 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001864189, 2024-09-17T16:01:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856328+0000", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3570000000000001e-06 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856466+0000", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137415 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856516+0000", 2024-09-17T16:01:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0541999999999998e-05 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856527+0000", 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4474999999999994e-05 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856611+0000", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0720000000000006e-06 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858143+0000", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015325499999999999 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858191+0000", 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000003e-05 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1768714789999999, 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=3, reqid=client.4560.0:11, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.174945589, 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096021099999999999, 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858875+0000", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0359999999999999e-06 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858897+0000", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1219000000000001e-05 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858943+0000", 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6718000000000002e-05 2024-09-17T16:01:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858952+0000", 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9449999999999999e-06 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859019+0000", 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6594999999999995e-05 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859029+0000", 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0327e-05 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859792+0000", 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000762655 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859835+0000", 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2716000000000002e-05 2024-09-17T16:01:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1739089869999999, 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:01:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.172483868, 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087109700000000004, 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861337+0000", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861358+0000", 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1438e-05 2024-09-17T16:01:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861402+0000", 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3157e-05 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861414+0000", 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2638e-05 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861509+0000", 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4903999999999998e-05 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861524+0000", 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4313e-05 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.862172+0000", 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064898699999999996 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.862207+0000", 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4761000000000001e-05 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1703338639999998, 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001910307, 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863487+0000", 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863535+0000", 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7713999999999998e-05 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863653+0000", 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117757 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863662+0000", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9779999999999994e-06 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863800+0000", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013886599999999999 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863813+0000", 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.295e-05 2024-09-17T16:01:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865357+0000", 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001543464 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865396+0000", 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9629000000000003e-05 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1702786569999999, 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:01:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:01:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1688446410000002, 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093210899999999998, 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864976+0000", 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9400000000000004e-07 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864995+0000", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8513999999999999e-05 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865053+0000", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8217999999999997e-05 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0042e-05 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865137+0000", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4350000000000005e-05 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865148+0000", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0400999999999999e-05 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865868+0000", 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071984700000000002 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865907+0000", 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9943e-05 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=7, reqid=client.4560.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1656076510000002, 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092475100000000002, 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868214+0000", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.192e-06 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868228+0000", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4003e-05 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868272+0000", 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4536999999999997e-05 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868283+0000", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0781999999999999e-05 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868373+0000", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0136000000000006e-05 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868384+0000", 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0808000000000001e-05 2024-09-17T16:01:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.869097+0000", 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071339000000000005 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.869137+0000", 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9903000000000001e-05 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=2, reqid=client.4560.0:61, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1491918330000002, 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018138760000000001, 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9e-08 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884974+0000", 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034541900000000002 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885013+0000", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000002e-05 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885023+0000", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885103+0000", 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9871999999999997e-05 2024-09-17T16:01:37.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885112+0000", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4769999999999992e-06 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886400+0000", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012878869999999999 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886442+0000", 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2202e-05 2024-09-17T16:01:37.099 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=3, reqid=client.4560.0:62, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.888431+0000", 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1453889369999999, 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00086782299999999999, 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888431+0000", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888431+0000", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888432+0000", 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888452+0000", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0089999999999999e-05 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888492+0000", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0428e-05 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888502+0000", 2024-09-17T16:01:37.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888574+0000", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2379000000000006e-05 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888587+0000", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2089000000000001e-05 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889243+0000", 2024-09-17T16:01:37.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065643500000000003 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889299+0000", 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5909999999999998e-05 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=11, reqid=client.4560.0:91, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1261072799999998, 2024-09-17T16:01:37.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001672984, 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:37.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907717+0000", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6290000000000001e-06 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907950+0000", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023220000000000001 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907988+0000", 2024-09-17T16:01:37.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8900999999999999e-05 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907997+0000", 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908086+0000", 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908108+0000", 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2447e-05 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909346+0000", 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001237782 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909386+0000", 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9554000000000001e-05 2024-09-17T16:01:37.108 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=12, reqid=client.4560.0:92, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1250036670000001, 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087827399999999996, 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908817+0000", 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.92e-07 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908843+0000", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6046000000000001e-05 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908884+0000", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1056000000000001e-05 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908894+0000", 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6330000000000008e-06 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908965+0000", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0817999999999999e-05 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908975+0000", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0587e-05 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909659+0000", 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068393700000000002 2024-09-17T16:01:37.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909695+0000", 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5404999999999997e-05 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=13, reqid=client.4560.0:94, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1225896880000001, 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012298319999999999, 2024-09-17T16:01:37.113 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911231+0000", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1200000000000002e-07 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911496+0000", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026446999999999999 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911523+0000", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7665e-05 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911529+0000", 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1530000000000002e-06 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911584+0000", 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4611000000000003e-05 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911592+0000", 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912421+0000", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082916399999999996 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912460+0000", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9047000000000001e-05 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=16, reqid=client.4560.0:95, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.918108+0000", 2024-09-17T16:01:37.117 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.1157116199999999, 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072609699999999998, 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918108+0000", 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918108+0000", 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918109+0000", 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4600000000000003e-07 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918142+0000", 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2697999999999998e-05 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918183+0000", 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0852999999999997e-05 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918195+0000", 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1619e-05 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918267+0000", 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2840000000000001e-05 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918277+0000", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3389999999999999e-06 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918801+0000", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000523873 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.918835+0000", 2024-09-17T16:01:37.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3929e-05 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=17, reqid=client.4560.0:124, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.0852009520000001, 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016306890000000001, 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:37.122 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948620+0000", 2024-09-17T16:01:37.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.238e-06 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949131+0000", 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051077299999999996 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949166+0000", 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4619999999999997e-05 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949174+0000", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2400000000000007e-06 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949224+0000", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0534000000000001e-05 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949231+0000", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.229e-06 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950221+0000", 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098999599999999993 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950250+0000", 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9059000000000001e-05 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.126 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:37.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:01:37.128 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:37.125+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:37.229 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:37.227+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:37.236 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:37.236 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:37.236 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:37.237 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:37.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:01:37.329 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:37.328+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:37.393 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:37.393 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:37.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:01:37.442 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:37.440+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:37.532 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:37.530+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:5 3.bs0 3:d4004f6b:::smithi18136759-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:37.552 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6969395899999999, 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010865848000000001, 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:37.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2099999999999997e-07 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852349+0000", 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3249999999999999e-06 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852356+0000", 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6040000000000004e-06 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852361+0000", 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.437e-06 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853722+0000", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136107 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854082+0000", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036069599999999998 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854755+0000", 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067232100000000003 2024-09-17T16:01:37.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856366+0000", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016111039999999999 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863131+0000", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067652729999999996 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863211+0000", 2024-09-17T16:01:37.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9497000000000001e-05 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.696917365, 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:01:37.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:19 3.1s0 3:8347559a:::smithi18136759-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6968749760000001, 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010852832999999999, 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.563 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 19 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:37.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852410+0000", 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852655+0000", 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024543600000000003 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852672+0000", 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6328e-05 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852682+0000", 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0365e-05 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852763+0000", 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0810999999999996e-05 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852923+0000", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016075700000000001 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853852+0000", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092845700000000005 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856526+0000", 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026744960000000002 2024-09-17T16:01:37.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863195+0000", 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066689100000000001 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863262+0000", 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6847000000000006e-05 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6965684059999999, 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.569 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:01:37.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:37.574 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6965282749999999, 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:01:37.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:46 3.6s0 3:6d55c691:::smithi18136759-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.685038112, 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010233671, 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 46 2024-09-17T16:01:37.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864248+0000", 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7400000000000001e-06 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865054+0000", 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000806429 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865069+0000", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4673e-05 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865078+0000", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6880000000000005e-06 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865530+0000", 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000452724 2024-09-17T16:01:37.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865698+0000", 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016729700000000001 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.866696+0000", 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099818099999999989 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868021+0000", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013246709999999999 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874396+0000", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063754190000000002 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874480+0000", 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3849000000000003e-05 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.683850981, 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.586 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:01:37.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:01:37.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:01:37.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.68357774, 2024-09-17T16:01:37.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:01:37.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:01:37.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:49 3.7s0 3:ec2b5851:::smithi18136759-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.596 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6811019759999999, 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098686610000000008, 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 49 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:01:37.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868183+0000", 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6700000000000002e-07 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868516+0000", 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033280000000000001 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868533+0000", 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7329e-05 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868542+0000", 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9700000000000005e-06 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868658+0000", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115648 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868779+0000", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012072499999999999 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.869537+0000", 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075870500000000003 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871073+0000", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001535916 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.877968+0000", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068943310000000001 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.878051+0000", 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3269999999999994e-05 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.601 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6807119579999998, 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:01:37.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:01:37.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:01:37.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6806600309999999, 2024-09-17T16:01:37.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:37.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:01:37.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:66 3.6s0 3:60a7d8b5:::smithi18136759-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.612 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6687200349999998, 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010929104, 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880565+0000", 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880568+0000", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1939999999999998e-06 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880578+0000", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-06 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880598+0000", 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9964999999999999e-05 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882449+0000", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018505730000000001 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882543+0000", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4135999999999995e-05 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890649+0000", 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081062559999999992 2024-09-17T16:01:37.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890748+0000", 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9234000000000006e-05 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891493+0000", 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074507600000000001 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6601192650000001, 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.618 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:01:37.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6600360910000003, 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.623 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:01:37.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.628 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.645593324, 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:01:37.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:01:37.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:37.634 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6455536340000001, 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:01:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:98 3.7s0 3:ec861466:::smithi18136759-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:37.639 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6357305330000003, 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099957889999999997, 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 98 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:37.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913555+0000", 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4700000000000001e-07 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913560+0000", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9969999999999998e-06 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913568+0000", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4740000000000005e-06 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913574+0000", 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7309999999999998e-06 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.914995+0000", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001421381 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915104+0000", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010908000000000001 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915686+0000", 2024-09-17T16:01:37.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058229899999999999 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916507+0000", 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082023600000000001 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923452+0000", 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069455970000000004 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923550+0000", 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7246999999999999e-05 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:99 3.7s0 3:ec861466:::smithi18136759-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6356919329999999, 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011511970999999999, 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 99 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913593+0000", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913596+0000", 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2140000000000001e-06 2024-09-17T16:01:37.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913605+0000", 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913612+0000", 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915969+0000", 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023571690000000001 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916049+0000", 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9777000000000005e-05 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924068+0000", 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080191269999999992 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924152+0000", 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4005999999999995e-05 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.925104+0000", 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095257899999999995 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:122 3.bs0 3:dee30a9d:::smithi18136759-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.650 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.6061218630000003, 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011179774, 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 122 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:37.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943163+0000", 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2200000000000004e-07 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943166+0000", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943176+0000", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.166e-06 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943181+0000", 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.502e-06 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944736+0000", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015548140000000001 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944814+0000", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7945e-05 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945207+0000", 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039347299999999998 2024-09-17T16:01:37.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945618+0000", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041055900000000001 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954241+0000", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.008622691 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954342+0000", 2024-09-17T16:01:37.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010164 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:123 3.bs0 3:dee30a9d:::smithi18136759-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.606093548, 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012212639000000001, 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:37.656 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 123 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:37.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943194+0000", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.565e-06 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943202+0000", 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0050000000000004e-06 2024-09-17T16:01:37.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943207+0000", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3260000000000003e-06 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945420+0000", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022128310000000002 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945469+0000", 2024-09-17T16:01:37.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8897999999999998e-05 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954714+0000", 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092452019999999992 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954800+0000", 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5541e-05 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:37.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.955403+0000", 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060376199999999998 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:37.661 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:37.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:01:37.663 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:37.661+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:37.764 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:37.762+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:37.778 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:37.778 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:37.778 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:37.779 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:37.787 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:01:37.864 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:37.862+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:37.869 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:01:37.869 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:01:37.874 INFO:tasks.rados.rados.0.smithi181.stdout:66: done (3 left) 2024-09-17T16:01:37.874 INFO:tasks.rados.rados.0.smithi181.stdout:67: copy_from oid 42 from oid 48 current snap is 3 2024-09-17T16:01:37.874 INFO:tasks.rados.rados.0.smithi181.stdout:63: finishing write tid 1 to smithi18136759-20 2024-09-17T16:01:37.874 INFO:tasks.rados.rados.0.smithi181.stdout:63: finishing write tid 2 to smithi18136759-20 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: finishing write tid 3 to smithi18136759-20 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: finishing write tid 4 to smithi18136759-20 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: oid 20 updating version 0 to 11 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: oid 20 updating version 11 to 12 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: oid 20 updating version 12 to 13 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 13 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: left oid 20 (ObjNum 54 snap 3 seq_num 54) 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:65: finishing rollback tid 1 to smithi18136759-45 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:63: done (3 left) 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:68: rollback oid 1 current snap is 3 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 1 to 0 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:65: finishing rollback tid 2 to smithi18136759-45 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-17T16:01:37.875 INFO:tasks.rados.rados.0.smithi181.stdout:64: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 14654 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:64: done (3 left) 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:65: done (2 left) 2024-09-17T16:01:37.876 INFO:tasks.rados.rados.0.smithi181.stdout:69: snap_create 2024-09-17T16:01:37.881 INFO:tasks.rados.rados.0.smithi181.stdout:68: finishing rollback tid 1 to smithi18136759-1 2024-09-17T16:01:37.882 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 0 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-17T16:01:37.890 INFO:tasks.rados.rados.0.smithi181.stdout:67: finishing copy_from racing read to smithi18136759-42 2024-09-17T16:01:37.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-17T16:01:37.893 INFO:tasks.rados.rados.0.smithi181.stdout:67: finishing copy_from to smithi18136759-42 2024-09-17T16:01:37.893 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 7 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T16:01:37.898 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:37.899 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:37.899 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:37.899 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:37.899 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:37.907 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:01:37.964 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:37.962+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.012 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:25 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'15, mlcod=24'11)", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.801058+0000", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.208404520999999, 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053455630000000002, 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801058+0000", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801058+0000", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801058+0000", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2900000000000003e-07 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801065+0000", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7070000000000003e-06 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801089+0000", 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3961999999999999e-05 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801095+0000", 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8610000000000003e-06 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801132+0000", 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7588e-05 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.801148+0000", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5449000000000001e-05 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.806364+0000", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052161559999999996 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.806403+0000", 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9110999999999999e-05 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.016 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:40 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'22, mlcod=25'19)", 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.086116031, 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069417209999999997, 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923347+0000", 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923352+0000", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7929999999999997e-06 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923384+0000", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2196000000000001e-05 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923394+0000", 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0369e-05 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923437+0000", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3046000000000001e-05 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923451+0000", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3556e-05 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930260+0000", 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068091169999999999 2024-09-17T16:01:38.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930288+0000", 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7937e-05 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:41 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'19)", 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.085577859000001, 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077219050000000003, 2024-09-17T16:01:38.021 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923885+0000", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923947+0000", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1790000000000003e-05 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923974+0000", 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7368999999999998e-05 2024-09-17T16:01:38.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923980+0000", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924022+0000", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924037+0000", 2024-09-17T16:01:38.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4127e-05 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931579+0000", 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075422340000000001 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931606+0000", 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7307000000000001e-05 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:42 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'24, mlcod=25'19)", 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.085055028999999, 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072286889999999999, 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924408+0000", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0699999999999997e-07 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924412+0000", 2024-09-17T16:01:38.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924432+0000", 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9870000000000001e-05 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924436+0000", 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6e-06 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924471+0000", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4527999999999999e-05 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924487+0000", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6303999999999998e-05 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931609+0000", 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071217429999999998 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931636+0000", 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6911e-05 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:43 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'25, mlcod=25'19)", 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.084483935000002, 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066863529999999999, 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.030 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924979+0000", 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924993+0000", 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4431e-05 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925018+0000", 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5038999999999999e-05 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925023+0000", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6229999999999996e-06 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925060+0000", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6758999999999998e-05 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925072+0000", 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.216e-05 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931638+0000", 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065666329999999997 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931664+0000", 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6052000000000001e-05 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:44 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'26, mlcod=25'20)", 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.083179960999999, 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054079590000000004, 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926283+0000", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926321+0000", 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7805000000000001e-05 2024-09-17T16:01:38.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926348+0000", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7566e-05 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926354+0000", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2290000000000002e-06 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926396+0000", 2024-09-17T16:01:38.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2391999999999997e-05 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926409+0000", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3298999999999999e-05 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931667+0000", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052574520000000001 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931690+0000", 2024-09-17T16:01:38.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3391999999999999e-05 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:45 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'27, mlcod=25'20)", 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.926694+0000", 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.082768241, 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050230209999999999, 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.039 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926694+0000", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926694+0000", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926694+0000", 2024-09-17T16:01:38.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7299999999999995e-07 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926700+0000", 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9429999999999997e-06 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926724+0000", 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3209000000000001e-05 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926728+0000", 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5839999999999996e-06 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926763+0000", 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5240999999999997e-05 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926775+0000", 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1627999999999999e-05 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931691+0000", 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049158989999999996 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931717+0000", 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5843999999999999e-05 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:48 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'30, mlcod=25'20)", 2024-09-17T16:01:38.043 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.080580222999998, 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055739780000000003, 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928898+0000", 2024-09-17T16:01:38.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6059999999999999e-05 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928923+0000", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4961999999999999e-05 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928928+0000", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.921e-06 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928968+0000", 2024-09-17T16:01:38.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9416999999999999e-05 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928981+0000", 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3548e-05 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934429+0000", 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054479539999999996 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934456+0000", 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6551999999999999e-05 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:49 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'31, mlcod=25'20)", 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.929497+0000", 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.079965504, 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049865389999999999, 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.048 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929497+0000", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929497+0000", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929497+0000", 2024-09-17T16:01:38.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1800000000000005e-07 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929503+0000", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5809999999999996e-06 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929531+0000", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8336000000000001e-05 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.929546+0000", 2024-09-17T16:01:38.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4679e-05 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930076+0000", 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00053039799999999996 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930090+0000", 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3541e-05 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.051 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934458+0000", 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043680969999999996 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934483+0000", 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5089e-05 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:82 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'60, mlcod=25'58)", 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.966747478999999, 2024-09-17T16:01:38.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064218920000000002, 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:38.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042718+0000", 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0879999999999999e-06 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042736+0000", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7912999999999999e-05 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042742+0000", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9669999999999998e-06 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042787+0000", 2024-09-17T16:01:38.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4610999999999997e-05 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042801+0000", 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.437e-05 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.049102+0000", 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063004239999999998 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.049136+0000", 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4958e-05 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:95 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'69, mlcod=25'66)", 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.138872+0000", 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.870589571, 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048070489999999999, 2024-09-17T16:01:38.057 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138872+0000", 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138872+0000", 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138873+0000", 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138882+0000", 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7369999999999995e-06 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138896+0000", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.488e-05 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138901+0000", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2710000000000003e-06 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138938+0000", 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7478000000000003e-05 2024-09-17T16:01:38.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.138952+0000", 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3478e-05 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.143643+0000", 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046911690000000002 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.143680+0000", 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6687000000000003e-05 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:98 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'72, mlcod=25'66)", 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.869015827999998, 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005965237, 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.062 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140447+0000", 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3e-07 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140454+0000", 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3030000000000002e-06 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140481+0000", 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6506e-05 2024-09-17T16:01:38.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140487+0000", 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9619999999999998e-06 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140519+0000", 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2567999999999997e-05 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140531+0000", 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1321999999999999e-05 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146386+0000", 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005855206 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146411+0000", 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5639999999999998e-05 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:99 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'73, mlcod=25'66)", 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.868559943000001, 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055288220000000001, 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140903+0000", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5200000000000002e-07 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140968+0000", 2024-09-17T16:01:38.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5388999999999993e-05 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140986+0000", 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7946999999999999e-05 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140992+0000", 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8200000000000002e-06 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.069 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141026+0000", 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4745e-05 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141040+0000", 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3997e-05 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146413+0000", 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053725379999999996 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.070 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146431+0000", 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7934000000000001e-05 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:100 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'74, mlcod=25'66)", 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.141597+0000", 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.867865058, 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048483550000000004, 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.071 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141597+0000", 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141597+0000", 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.072 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141597+0000", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9799999999999999e-07 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141603+0000", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5829999999999997e-06 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141619+0000", 2024-09-17T16:01:38.073 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5971e-05 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141625+0000", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3559999999999998e-06 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141656+0000", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0491e-05 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141669+0000", 2024-09-17T16:01:38.074 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2751e-05 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146432+0000", 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047632229999999996 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146445+0000", 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3582e-05 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.075 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.123166651, 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053513140000000002, 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.076 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886296+0000", 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1299999999999999e-07 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.077 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5797000000000001e-05 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0488999999999999e-05 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886349+0000", 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3929999999999996e-06 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.078 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886417+0000", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8350999999999997e-05 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886428+0000", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0397000000000001e-05 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891609+0000", 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051809150000000003 2024-09-17T16:01:38.079 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891647+0000", 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8158999999999998e-05 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887103+0000", 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.1223595529999999, 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045707389999999999, 2024-09-17T16:01:38.080 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887103+0000", 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887103+0000", 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.081 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887103+0000", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887120+0000", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7142e-05 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887154+0000", 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3358999999999997e-05 2024-09-17T16:01:38.082 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887162+0000", 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4190000000000002e-06 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887225+0000", 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3057000000000003e-05 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.083 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887231+0000", 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.784e-06 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891652+0000", 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004420429 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891673+0000", 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1702999999999999e-05 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.084 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 39/32 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=39'20, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.877984+0000", 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.13147814099999999, 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052730980000000004, 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.085 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.877984+0000", 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.877984+0000", 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.877985+0000", 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0800000000000004e-07 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878002+0000", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7779e-05 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878032+0000", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9413000000000001e-05 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878040+0000", 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1960000000000008e-06 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878095+0000", 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4703000000000001e-05 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878104+0000", 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.431e-06 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.883227+0000", 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051227729999999997 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.883257+0000", 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0195000000000001e-05 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(osd.14.22:1 3.es0 3:7408d595:::smithi18136759-48:head [assert-version v7,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:38.089 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.129991994, 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059963730000000002, 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:38.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879471+0000", 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3170000000000001e-06 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879476+0000", 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5839999999999996e-06 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879515+0000", 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8578000000000002e-05 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879519+0000", 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5059999999999997e-06 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879568+0000", 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8609000000000002e-05 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879647+0000", 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9402999999999999e-05 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.885466+0000", 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058193760000000002 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.093 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.3s4 39/32 ECSubWrite(tid=13, reqid=osd.15.0:13, at_version=39'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.881007+0000", 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.128454705, 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052804649999999998, 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881007+0000", 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881007+0000", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881008+0000", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8500000000000002e-07 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881029+0000", 2024-09-17T16:01:38.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1242000000000001e-05 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881045+0000", 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6433999999999999e-05 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881053+0000", 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5109999999999997e-06 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.096 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881105+0000", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1780999999999998e-05 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.881110+0000", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3700000000000003e-06 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.886252+0000", 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051416960000000003 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.886288+0000", 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6146000000000003e-05 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 39/32 ECSubWrite(tid=158, reqid=osd.11.0:157, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.884219+0000", 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.12524268899999999, 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051239579999999996, 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.098 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884219+0000", 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884219+0000", 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884220+0000", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0200000000000002e-07 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884228+0000", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6450000000000005e-06 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884249+0000", 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1214999999999999e-05 2024-09-17T16:01:38.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884263+0000", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4344e-05 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884313+0000", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9675999999999997e-05 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884322+0000", 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.329e-06 2024-09-17T16:01:38.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889306+0000", 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049837889999999998 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889343+0000", 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7357999999999997e-05 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.102 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.104 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:01:38.104 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:38.102+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:38.205 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:38.203+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:38.207 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.207 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:38.208 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:38.208 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:01:38.306 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:38.304+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:38.320 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.320 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:38.320 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:38.320 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:38.320 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.329 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:01:38.407 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:38.405+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:38.432 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.432 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 34/32 ECSubWrite(tid=113, reqid=client.4560.0:1, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852552+0000", 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5775655630000003, 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018770029999999999, 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852552+0000", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852552+0000", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852554+0000", 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.39e-06 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852606+0000", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2244999999999999e-05 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852649+0000", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3590000000000001e-05 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852661+0000", 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.182e-05 2024-09-17T16:01:38.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852747+0000", 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5908999999999994e-05 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852760+0000", 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3311e-05 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854379+0000", 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001618399 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854429+0000", 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0339e-05 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 34/32 ECSubWrite(tid=0, reqid=client.4560.0:4, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.853222+0000", 2024-09-17T16:01:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5768960070000002, 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00143969, 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853222+0000", 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853222+0000", 2024-09-17T16:01:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853223+0000", 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2270000000000001e-06 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853264+0000", 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0725000000000001e-05 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853310+0000", 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6690000000000002e-05 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853321+0000", 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0186999999999999e-05 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853394+0000", 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3427e-05 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853404+0000", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0383e-05 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854619+0000", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001214131 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854661+0000", 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2920000000000002e-05 2024-09-17T16:01:38.441 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 34/32 ECSubWrite(tid=115, reqid=client.4560.0:2, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854014+0000", 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.5761034040000004, 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012557340000000001, 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854014+0000", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854014+0000", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854015+0000", 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3099999999999996e-07 2024-09-17T16:01:38.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854036+0000", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0747999999999999e-05 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854084+0000", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8347999999999999e-05 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854095+0000", 2024-09-17T16:01:38.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0681e-05 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854173+0000", 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8203999999999999e-05 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854183+0000", 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855233+0000", 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010501429999999999 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855270+0000", 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6702999999999997e-05 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=145, reqid=client.4560.0:174, at_version=38'9, trim_to=0'0, roll_forward_to=35'7))", 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.869779+0000", 2024-09-17T16:01:38.446 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.560338733, 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0014276320000000001, 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869779+0000", 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869779+0000", 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869779+0000", 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9000000000000001e-07 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869806+0000", 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6601000000000002e-05 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869863+0000", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7606999999999999e-05 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869875+0000", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1861999999999999e-05 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869969+0000", 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3614000000000002e-05 2024-09-17T16:01:38.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.869984+0000", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4674e-05 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.871159+0000", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001175618 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.871207+0000", 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7466000000000003e-05 2024-09-17T16:01:38.450 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=147, reqid=client.4560.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.555654315, 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022903149999999998, 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874464+0000", 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9599999999999998e-07 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874479+0000", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4235e-05 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874516+0000", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6990000000000003e-05 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874524+0000", 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6370000000000005e-06 2024-09-17T16:01:38.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874587+0000", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2878000000000003e-05 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874597+0000", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0088000000000001e-05 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.876717+0000", 2024-09-17T16:01:38.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002119606 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.876754+0000", 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6984999999999999e-05 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=148, reqid=client.4560.0:175, at_version=38'11, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.881153+0000", 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5489647689999999, 2024-09-17T16:01:38.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021494019999999999, 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881153+0000", 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881153+0000", 2024-09-17T16:01:38.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881154+0000", 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881635+0000", 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048159599999999999 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881667+0000", 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1854999999999999e-05 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881676+0000", 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.636e-06 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881736+0000", 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9827999999999997e-05 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.881745+0000", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6139999999999998e-06 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883264+0000", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015190609999999999 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883302+0000", 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8041999999999999e-05 2024-09-17T16:01:38.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=149, reqid=client.4560.0:176, at_version=38'12, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.883751+0000", 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.546366881, 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00172624, 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883751+0000", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883751+0000", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883752+0000", 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1200000000000001e-06 2024-09-17T16:01:38.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883950+0000", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000198261 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883989+0000", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8745999999999998e-05 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.883997+0000", 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2770000000000005e-06 2024-09-17T16:01:38.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884061+0000", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3847e-05 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884070+0000", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1849999999999993e-06 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.885437+0000", 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013664199999999999 2024-09-17T16:01:38.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.885477+0000", 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0383999999999998e-05 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5454331799999999, 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038886110000000002, 2024-09-17T16:01:38.464 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884707+0000", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2036999999999999e-05 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884745+0000", 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7843999999999999e-05 2024-09-17T16:01:38.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884756+0000", 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1240000000000001e-05 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884826+0000", 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9873e-05 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884836+0000", 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0053e-05 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888486+0000", 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036497270000000002 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888573+0000", 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7068000000000001e-05 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.468 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.543823385, 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:01:38.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:01:38.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:01:38.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.543012142, 2024-09-17T16:01:38.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044954000000000001, 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4892000000000002e-05 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887167+0000", 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5617999999999997e-05 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887176+0000", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6379999999999993e-06 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887237+0000", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1760000000000005e-05 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887246+0000", 2024-09-17T16:01:38.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891572+0000", 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004325626 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891601+0000", 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9105e-05 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=154, reqid=client.4560.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.892252+0000", 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5378653839999998, 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00053619099999999999, 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892252+0000", 2024-09-17T16:01:38.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892252+0000", 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892253+0000", 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3500000000000005e-07 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892262+0000", 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2329999999999995e-06 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892293+0000", 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0648999999999997e-05 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892301+0000", 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9160000000000001e-06 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892351+0000", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0198000000000003e-05 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892357+0000", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8649999999999998e-06 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892756+0000", 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00039872699999999998 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.892789+0000", 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2768000000000002e-05 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "pg_update_log_missing(3.8s6 epoch 39/32 rep_tid 155 entries 39'16 (0'0) error 3:19aefe3f:::smithi18136759-1:head by client.4560.0:187 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 38'14)", 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.875939+0000", 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.55417901599999997, 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052229299999999995, 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.482 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.875939+0000", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.875939+0000", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.875939+0000", 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9299999999999999e-07 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.875946+0000", 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3189999999999997e-06 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876004+0000", 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7661000000000002e-05 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876020+0000", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5928999999999999e-05 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876113+0000", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3003000000000004e-05 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876461+0000", 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00034808799999999998 2024-09-17T16:01:38.485 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 39/32 ECSubWrite(tid=156, reqid=osd.11.0:156, at_version=39'18, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.876943+0000", 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.55317505300000003, 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016876440000000001, 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876943+0000", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876943+0000", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876943+0000", 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:01:38.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876963+0000", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9879999999999999e-05 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876988+0000", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5219999999999999e-05 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.876996+0000", 2024-09-17T16:01:38.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2710000000000003e-06 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.877045+0000", 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9815000000000003e-05 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.877054+0000", 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3960000000000006e-06 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878598+0000", 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015443659999999999 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878630+0000", 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2109e-05 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 39/32 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=39'20, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:38.490 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.878001+0000", 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.55211645799999998, 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002212814, 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878001+0000", 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878001+0000", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878002+0000", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5199999999999996e-07 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878023+0000", 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1129999999999999e-05 2024-09-17T16:01:38.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878052+0000", 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9289e-05 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878060+0000", 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8120000000000008e-06 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878118+0000", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7635e-05 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.878127+0000", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9230000000000008e-06 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.880192+0000", 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002065061 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.880214+0000", 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2311999999999999e-05 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 39/32 ECSubWrite(tid=158, reqid=osd.11.0:157, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.884244+0000", 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.54587363099999997, 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00075451700000000001, 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.495 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884244+0000", 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884244+0000", 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884245+0000", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.78e-07 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884254+0000", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2220000000000008e-06 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884281+0000", 2024-09-17T16:01:38.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7016999999999999e-05 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884288+0000", 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9609999999999999e-06 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884340+0000", 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1811999999999998e-05 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884349+0000", 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9910000000000008e-06 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884958+0000", 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00060915199999999996 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.884999+0000", 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0484000000000001e-05 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.499 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.500 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:01:38.545 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:38.543+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:38.605 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.605 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:38.605 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:38.605 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.607 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:38.606+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:38.613 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:01:38.708 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:38.706+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:38.757 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.758 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:38.758 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:38.758 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:38.758 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.766 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:01:38.809 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:38.807+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:38.870 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:38.870 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:38.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:38.870 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0126237180000004, 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:38.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:01:38.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:01:38.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:38.875 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0117228440000003, 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:01:38.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.856618+0000", 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0109881679999999, 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021061119999999998, 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.880 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856618+0000", 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856618+0000", 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856618+0000", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8000000000000001e-07 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856693+0000", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4839999999999995e-05 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856743+0000", 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9230999999999997e-05 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856753+0000", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0456e-05 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856902+0000", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014879100000000001 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856930+0000", 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7943e-05 2024-09-17T16:01:38.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858675+0000", 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001744949 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858724+0000", 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9521999999999998e-05 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0076330169999999, 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:38.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:01:38.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:38.889 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0062945699999997, 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055595449999999999, 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861313+0000", 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9800000000000002e-07 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861333+0000", 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0276999999999999e-05 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861372+0000", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9365e-05 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861382+0000", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3349999999999996e-06 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861456+0000", 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3972000000000003e-05 2024-09-17T16:01:38.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861465+0000", 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7480000000000005e-06 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866841+0000", 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053751720000000001 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866871+0000", 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0678e-05 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0053294700000004, 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045470969999999999, 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.894 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862278+0000", 2024-09-17T16:01:38.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5199999999999995e-07 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862373+0000", 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5013999999999995e-05 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862431+0000", 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862440+0000", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6339999999999996e-06 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862513+0000", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3028999999999995e-05 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862522+0000", 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866779+0000", 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042571140000000002 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866824+0000", 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4511000000000001e-05 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.898 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0052260559999997, 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:01:38.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:01:38.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0034006780000002, 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.903 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:38.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.907 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.003252603, 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:01:38.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:01:38.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0026027190000004, 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.912 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:01:38.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=6, reqid=client.4560.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0000451799999999, 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041413530000000004, 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867562+0000", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6599999999999994e-07 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867576+0000", 2024-09-17T16:01:38.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4229999999999999e-05 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867702+0000", 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012607699999999999 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867712+0000", 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867779+0000", 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7217000000000004e-05 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867789+0000", 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6700000000000006e-06 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871665+0000", 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003875898 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871702+0000", 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7444000000000003e-05 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.0000253560000001, 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.921 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:01:38.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9995341839999998, 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:01:38.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:01:38.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9994086859999998, 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035317220000000002, 2024-09-17T16:01:38.930 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868208+0000", 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6989999999999998e-06 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868243+0000", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5157000000000002e-05 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868250+0000", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1489999999999997e-06 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868323+0000", 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2859000000000002e-05 2024-09-17T16:01:38.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868334+0000", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0601000000000001e-05 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871706+0000", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033717389999999999 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871729+0000", 2024-09-17T16:01:38.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3648000000000001e-05 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=6, reqid=client.4560.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9987973329999997, 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048136530000000002, 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.935 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868810+0000", 2024-09-17T16:01:38.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9099999999999991e-07 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868951+0000", 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014129799999999999 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868987+0000", 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5556000000000003e-05 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868996+0000", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6549999999999999e-06 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869062+0000", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5714000000000001e-05 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869072+0000", 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0273e-05 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873569+0000", 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044970189999999997 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873623+0000", 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3146999999999998e-05 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.939 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=7, reqid=client.4560.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.871822+0000", 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9957843620000002, 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022419010000000001, 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871822+0000", 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871822+0000", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871823+0000", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8599999999999997e-07 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871843+0000", 2024-09-17T16:01:38.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0126999999999999e-05 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871913+0000", 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0308000000000005e-05 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871921+0000", 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2840000000000006e-06 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871993+0000", 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1784000000000003e-05 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.872002+0000", 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0489999999999993e-06 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.874029+0000", 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002026838 2024-09-17T16:01:38.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.874064+0000", 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4625000000000001e-05 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=12, reqid=client.4560.0:26, at_version=0'0, trim_to=0'0, roll_forward_to=34'4))", 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.878085+0000", 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9895216290000004, 2024-09-17T16:01:38.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00192439, 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878085+0000", 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878085+0000", 2024-09-17T16:01:38.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878085+0000", 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6099999999999999e-07 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878108+0000", 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2612e-05 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878146+0000", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7898000000000002e-05 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878156+0000", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0421999999999999e-05 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878238+0000", 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1345000000000001e-05 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.878248+0000", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0268999999999999e-05 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879965+0000", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017174480000000001 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880009+0000", 2024-09-17T16:01:38.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3535000000000002e-05 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=17, reqid=client.4560.0:124, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.948606+0000", 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.9190002870000002, 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0016522360000000001, 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948606+0000", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948606+0000", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948608+0000", 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.705e-06 2024-09-17T16:01:38.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948780+0000", 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00017197 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948833+0000", 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3749999999999999e-05 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948843+0000", 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5550000000000008e-06 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948912+0000", 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9122000000000007e-05 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.948919+0000", 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.81e-06 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950222+0000", 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013034699999999999 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950258+0000", 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5853999999999999e-05 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.953 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=44, reqid=client.4560.0:89, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.977415+0000", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 4.8901914770000001, 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0019134829999999999, 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977415+0000", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977415+0000", 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977416+0000", 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2389999999999999e-06 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977430+0000", 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3752e-05 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977461+0000", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1658000000000001e-05 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977469+0000", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5719999999999997e-06 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977536+0000", 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6765000000000002e-05 2024-09-17T16:01:38.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.977548+0000", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2311e-05 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.979295+0000", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017472220000000001 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.979328+0000", 2024-09-17T16:01:38.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2963999999999998e-05 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.887787+0000", 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.97981928200000001, 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0020234789999999999, 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:38.958 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887787+0000", 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887787+0000", 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887789+0000", 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9530000000000002e-06 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887996+0000", 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00020757100000000001 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.888056+0000", 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9089000000000003e-05 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.888066+0000", 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0702e-05 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.888163+0000", 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7136e-05 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.888192+0000", 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8739999999999999e-05 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889769+0000", 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001576626 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889810+0000", 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1662000000000002e-05 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:38.964 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:38.964 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:38.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:38.965 INFO:tasks.rados.rados.0.smithi181.stdout:67: done (2 left) 2024-09-17T16:01:38.966 INFO:tasks.rados.rados.0.smithi181.stdout:68: done (1 left) 2024-09-17T16:01:38.966 INFO:tasks.rados.rados.0.smithi181.stdout:69: done (0 left) 2024-09-17T16:01:38.966 INFO:tasks.rados.rados.0.smithi181.stdout:70: snap_create 2024-09-17T16:01:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:01:38.967 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:01:38.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-17T16:01:38.968 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:38.963+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:39.066 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:39.064+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:39.112 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:39.113 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:39.113 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:39.113 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:39.121 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:01:39.156 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:39.156 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:39.167 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:39.165+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:39.179 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:39.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:39.200 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:39.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:39.265 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:39.265 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:39.265 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:39.265 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:39.265 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:39.267 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:39.266+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:39.273 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:01:39.368 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:39.366+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:7 3.0s0 3:03e99693:::smithi18136759-3:head [append 0~49152,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852510+0000", 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5237501760000001, 2024-09-17T16:01:39.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010430793000000001, 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 7 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852510+0000", 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852510+0000", 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852512+0000", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852668+0000", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000156813 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852685+0000", 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6395000000000001e-05 2024-09-17T16:01:39.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852696+0000", 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0949e-05 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852777+0000", 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1262999999999998e-05 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852960+0000", 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018339300000000001 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.853802+0000", 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00084157700000000004 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855698+0000", 2024-09-17T16:01:39.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018959370000000001 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862848+0000", 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007150188 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862941+0000", 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2517999999999997e-05 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:8 3.0s0 3:03e99693:::smithi18136759-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5235169290000004, 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011147927, 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.386 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 8 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852744+0000", 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.159e-06 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852751+0000", 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9120000000000001e-06 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852762+0000", 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1192000000000001e-05 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4889999999999998e-06 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854588+0000", 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018183909999999999 2024-09-17T16:01:39.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854806+0000", 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217347 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855286+0000", 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048023199999999998 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857505+0000", 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022190869999999998 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863759+0000", 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062543 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863891+0000", 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013181799999999999 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5234765909999997, 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.392 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:01:39.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:39.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:01:39.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5234480780000004, 2024-09-17T16:01:39.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:39.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:01:39.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:01:39.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.403 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5233090330000003, 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:01:39.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:01:39.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:39.409 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5232813590000003, 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:01:39.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 35/32 ECSubWrite(tid=0, reqid=client.4560.0:22, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.857506+0000", 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5187537520000003, 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096535930000000002, 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.415 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857506+0000", 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857506+0000", 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857506+0000", 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1299999999999999e-07 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857623+0000", 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117036 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857683+0000", 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9404000000000002e-05 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857691+0000", 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5520000000000005e-06 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857742+0000", 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0949999999999998e-05 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857752+0000", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6709999999999994e-06 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867122+0000", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0093703339999999993 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867160+0000", 2024-09-17T16:01:39.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7432999999999999e-05 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs2 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5174741709999999, 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012875312, 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:39.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6400000000000001e-07 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858933+0000", 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014714799999999999 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858998+0000", 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5139000000000001e-05 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859014+0000", 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5814000000000001e-05 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859094+0000", 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9672000000000006e-05 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859104+0000", 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0596e-05 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871628+0000", 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012524086 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871661+0000", 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2693e-05 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.424 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:39 3.9s0 3:905994b3:::smithi18136759-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5173015090000002, 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013438131000000001, 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 39 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858959+0000", 2024-09-17T16:01:39.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3399999999999998e-07 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858963+0000", 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.427e-06 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858970+0000", 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5270000000000001e-06 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858977+0000", 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9399999999999996e-06 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861610+0000", 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026333149999999998 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861663+0000", 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2731e-05 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868368+0000", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067046069999999996 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868428+0000", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0503e-05 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.872397+0000", 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039682470000000003 2024-09-17T16:01:39.429 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=2, last_complete=34'3, committed=1, applied=1))", 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5127051150000002, 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014226245, 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:39.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863557+0000", 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1169999999999998e-06 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863638+0000", 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.03e-05 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863644+0000", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4150000000000004e-06 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877622+0000", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013978305 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877781+0000", 2024-09-17T16:01:39.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015883299999999999 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5127014450000003, 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014337127, 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863559+0000", 2024-09-17T16:01:39.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5800000000000001e-07 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863560+0000", 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863674+0000", 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011379000000000001 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863682+0000", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877875+0000", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014193012 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877896+0000", 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0993e-05 2024-09-17T16:01:39.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s4 35/32 ECSubWrite(tid=119, reqid=client.4560.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5115626239999997, 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015598302, 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864698+0000", 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.44e-07 2024-09-17T16:01:39.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864708+0000", 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.73e-06 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864735+0000", 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7804e-05 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864741+0000", 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7969999999999997e-06 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877824+0000", 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01308231 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877833+0000", 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4809999999999995e-06 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880272+0000", 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024384799999999998 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880296+0000", 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4056000000000001e-05 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.442 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4588447809999998, 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.443 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:01:39.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.448 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4588195769999999, 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:01:39.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.453 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4554535829999997, 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:01:39.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.457 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4284789340000001, 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.458 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:01:39.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:01:39.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4284610630000003, 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.464 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:39.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:01:39.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4251672959999997, 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.469 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:01:39.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:185 3.9s0 3:99c5a4ac:::smithi18136759-42:head [copy-from ver 7] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.5033290880000001, 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019207894999999999, 2024-09-17T16:01:39.473 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 185 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872932+0000", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.635e-06 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872938+0000", 2024-09-17T16:01:39.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1680000000000003e-06 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872948+0000", 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0477e-05 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872955+0000", 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7750000000000004e-06 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873015+0000", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873024+0000", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9439999999999994e-06 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874582+0000", 2024-09-17T16:01:39.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00155798 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874682+0000", 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100215 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887716+0000", 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013033741999999999 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889595+0000", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018794879999999999 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892089+0000", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024933450000000001 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892139+0000", 2024-09-17T16:01:39.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0172e-05 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:186 3.9s0 3:99c5a4ac:::smithi18136759-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.503292453, 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016243253999999999, 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:39.480 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 186 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:39.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872968+0000", 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2500000000000001e-07 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872969+0000", 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.525e-06 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872975+0000", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.186e-06 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872978+0000", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.202e-06 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873075+0000", 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7590999999999994e-05 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873079+0000", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.54e-06 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874867+0000", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017885100000000001 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for blocked object", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874930+0000", 2024-09-17T16:01:39.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2175999999999996e-05 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889013+0000", 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014083304 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889112+0000", 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8972e-05 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:39.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889211+0000", 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8923000000000001e-05 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:39.486 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:39.487 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:39.488 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:39.488 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:39.488 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:39.489 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:01:39.489 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:39.490 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:39.485+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:39.490 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:39.490 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:39.509 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:39.509 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:39.529 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:39.551 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:39.551 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:39.573 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:39.573 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:39.588 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:39.586+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:39.594 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:39.594 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:39.604 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:39.604 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:39.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:01:39.614 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:39.614 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:39.636 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:39.637 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:39.637 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:39.634+0000 7fa2ed6f8640 0 -- 172.21.15.6:0/3232272662 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7fa2e82219e0 msgr2=0x7fa2e8223e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:39.689 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:39.688+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:39.759 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:39.759 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:39.767 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:01:39.790 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:39.788+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:39.858 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:39.859 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:39.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:01:39.881 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:39.881 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:39.882 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:01:39.890 INFO:tasks.rados.rados.0.smithi181.stdout:70: done (0 left) 2024-09-17T16:01:39.890 INFO:tasks.rados.rados.0.smithi181.stdout:71: read oid 25 snap -1 2024-09-17T16:01:39.890 INFO:tasks.rados.rados.0.smithi181.stdout:71: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T16:01:39.890 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 5 2024-09-17T16:01:39.890 INFO:tasks.rados.rados.0.smithi181.stdout:72: seq_num 55 ranges {393216=557056,950272=376832} 2024-09-17T16:01:39.891 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:39.888+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:39.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:01:39.895 INFO:tasks.rados.rados.0.smithi181.stdout:72: writing smithi18136759-24 from 393216 to 950272 tid 1 2024-09-17T16:01:39.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-17T16:01:39.898 INFO:tasks.rados.rados.0.smithi181.stdout:72: writing smithi18136759-24 from 950272 to 1327104 tid 2 2024-09-17T16:01:39.898 INFO:tasks.rados.rados.0.smithi181.stdout:73: snap_create 2024-09-17T16:01:39.898 INFO:tasks.rados.rados.0.smithi181.stdout:71: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T16:01:39.898 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:39.904 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 397367 2024-09-17T16:01:39.904 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:39.907 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing write tid 1 to smithi18136759-24 2024-09-17T16:01:39.908 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing write tid 2 to smithi18136759-24 2024-09-17T16:01:39.908 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing write tid 3 to smithi18136759-24 2024-09-17T16:01:39.909 INFO:tasks.rados.rados.0.smithi181.stdout:72: finishing write tid 4 to smithi18136759-24 2024-09-17T16:01:39.909 INFO:tasks.rados.rados.0.smithi181.stdout:72: oid 24 updating version 0 to 16 2024-09-17T16:01:39.910 INFO:tasks.rados.rados.0.smithi181.stdout:72: oid 24 version 16 is already newer than 15 2024-09-17T16:01:39.910 INFO:tasks.rados.rados.0.smithi181.stdout:72: oid 24 updating version 16 to 17 2024-09-17T16:01:39.910 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 17 (ObjNum 55 snap 5 seq_num 55) dirty exists 2024-09-17T16:01:39.910 INFO:tasks.rados.rados.0.smithi181.stdout:72: left oid 24 (ObjNum 55 snap 5 seq_num 55) 2024-09-17T16:01:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:39.991 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:39.989+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:39.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-17T16:01:40.092 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:40.090+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:40.136 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.137 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.145 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-17T16:01:40.193 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:40.191+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:40.247 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.248 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:01:40.295 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:40.293+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:40.395 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:40.393+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:40.398 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.398 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:01:40.497 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:40.494+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:40.509 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.509 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:01:40.598 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:40.597+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:40.661 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.661 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:01:40.699 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:40.697+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:40.773 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.773 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:01:40.800 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:40.798+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:40.884 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:40.884 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:40.891 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:01:40.891 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:01:40.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:71: done (2 left) 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:72: done (1 left) 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:73: done (0 left) 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:74: rollback oid 1 current snap is 6 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 1 to 1 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 9 current snap is 6 2024-09-17T16:01:40.897 INFO:tasks.rados.rados.0.smithi181.stdout:75: seq_num 56 ranges {147456=573440,720896=376832} 2024-09-17T16:01:40.901 INFO:tasks.rados.rados.0.smithi181.stdout:75: writing smithi18136759-9 from 147456 to 720896 tid 1 2024-09-17T16:01:40.901 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:40.899+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:40.903 INFO:tasks.rados.rados.0.smithi181.stdout:75: writing smithi18136759-9 from 720896 to 1097728 tid 2 2024-09-17T16:01:40.903 INFO:tasks.rados.rados.0.smithi181.stdout:76: delete oid 11 current snap is 6 2024-09-17T16:01:40.903 INFO:tasks.rados.rados.0.smithi181.stdout:74: finishing rollback tid 1 to smithi18136759-1 2024-09-17T16:01:40.904 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 0 (ObjNum 33 snap 0 seq_num 1476089744) dirty dne 2024-09-17T16:01:40.913 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing write tid 1 to smithi18136759-9 2024-09-17T16:01:40.914 INFO:tasks.rados.rados.0.smithi181.stdout:74: done (2 left) 2024-09-17T16:01:40.914 INFO:tasks.rados.rados.0.smithi181.stdout:76: done (1 left) 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:77: read oid 50 snap -1 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:77: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:78: read oid 27 snap -1 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:78: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 43 current snap is 6 2024-09-17T16:01:40.915 INFO:tasks.rados.rados.0.smithi181.stdout:79: seq_num 58 ranges {720896=606208,1327104=376832} 2024-09-17T16:01:40.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-17T16:01:40.922 INFO:tasks.rados.rados.0.smithi181.stdout:79: writing smithi18136759-43 from 720896 to 1327104 tid 1 2024-09-17T16:01:40.924 INFO:tasks.rados.rados.0.smithi181.stdout:79: writing smithi18136759-43 from 1327104 to 1703936 tid 2 2024-09-17T16:01:40.924 INFO:tasks.rados.rados.0.smithi181.stdout:80: snap_create 2024-09-17T16:01:40.924 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing write tid 2 to smithi18136759-9 2024-09-17T16:01:40.924 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing write tid 3 to smithi18136759-9 2024-09-17T16:01:40.925 INFO:tasks.rados.rados.0.smithi181.stdout:78: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T16:01:40.925 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.929 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.933 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.937 INFO:tasks.rados.rados.0.smithi181.stdout:77: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T16:01:40.937 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.945 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.953 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:40.960 INFO:tasks.rados.rados.0.smithi181.stdout:79: finishing write tid 1 to smithi18136759-43 2024-09-17T16:01:40.961 INFO:tasks.rados.rados.0.smithi181.stdout:79: finishing write tid 2 to smithi18136759-43 2024-09-17T16:01:41.000 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.000 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.002 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:40.999+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:41.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:01:41.103 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:41.101+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:41.120 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.120 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:01:41.204 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:41.202+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:41.213 INFO:tasks.rados.rados.0.smithi181.stdout:79: finishing write tid 3 to smithi18136759-43 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:79: finishing write tid 4 to smithi18136759-43 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:79: oid 43 updating version 0 to 8 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:79: oid 43 updating version 8 to 10 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:79: oid 43 version 10 is already newer than 9 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 10 (ObjNum 58 snap 6 seq_num 58) dirty exists 2024-09-17T16:01:41.215 INFO:tasks.rados.rados.0.smithi181.stdout:79: left oid 43 (ObjNum 58 snap 6 seq_num 58) 2024-09-17T16:01:41.237 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.238 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:01:41.305 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:41.303+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:41.355 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.355 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:01:41.405 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:41.403+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:41.471 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.471 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:01:41.524 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:41.522+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:41.550 INFO:tasks.rados.rados.0.smithi181.stdout:75: finishing write tid 4 to smithi18136759-9 2024-09-17T16:01:41.551 INFO:tasks.rados.rados.0.smithi181.stdout:75: oid 9 updating version 0 to 6 2024-09-17T16:01:41.551 INFO:tasks.rados.rados.0.smithi181.stdout:75: oid 9 updating version 6 to 7 2024-09-17T16:01:41.551 INFO:tasks.rados.rados.0.smithi181.stdout:75: oid 9 updating version 7 to 8 2024-09-17T16:01:41.551 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 8 (ObjNum 56 snap 6 seq_num 56) dirty exists 2024-09-17T16:01:41.551 INFO:tasks.rados.rados.0.smithi181.stdout:75: left oid 9 (ObjNum 56 snap 6 seq_num 56) 2024-09-17T16:01:41.587 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.587 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:01:41.607 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:41.605+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:41.709 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:41.707+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:41.744 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.744 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:01:41.809 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:41.808+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:41.861 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:41.861 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:41.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:01:41.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:01:41.895 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:01:41.898 INFO:tasks.rados.rados.0.smithi181.stdout:75: done (4 left) 2024-09-17T16:01:41.898 INFO:tasks.rados.rados.0.smithi181.stdout:77: done (3 left) 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:78: done (2 left) 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:79: done (1 left) 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:80: done (0 left) 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:81: copy_from oid 5 from oid 50 current snap is 7 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:82: copy_from oid 6 from oid 28 current snap is 7 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:83: setattr oid 48 current snap is 7 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 44 current snap is 7 2024-09-17T16:01:41.899 INFO:tasks.rados.rados.0.smithi181.stdout:84: seq_num 59 ranges {737280=622592,1359872=376832} 2024-09-17T16:01:41.905 INFO:tasks.rados.rados.0.smithi181.stdout:84: writing smithi18136759-44 from 737280 to 1359872 tid 1 2024-09-17T16:01:41.908 INFO:tasks.rados.rados.0.smithi181.stdout:84: writing smithi18136759-44 from 1359872 to 1736704 tid 2 2024-09-17T16:01:41.908 INFO:tasks.rados.rados.0.smithi181.stdout:85: snap_create 2024-09-17T16:01:41.908 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T16:01:41.910 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:41.908+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:41.911 INFO:tasks.rados.rados.0.smithi181.stdout:81: finishing copy_from racing read to smithi18136759-5 2024-09-17T16:01:41.912 INFO:tasks.rados.rados.0.smithi181.stdout:82: finishing copy_from racing read to smithi18136759-6 2024-09-17T16:01:41.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-17T16:01:41.916 INFO:tasks.rados.rados.0.smithi181.stdout:84: finishing write tid 1 to smithi18136759-44 2024-09-17T16:01:41.918 INFO:tasks.rados.rados.0.smithi181.stdout:84: finishing write tid 2 to smithi18136759-44 2024-09-17T16:01:41.918 INFO:tasks.rados.rados.0.smithi181.stdout:84: finishing write tid 3 to smithi18136759-44 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:84: finishing write tid 4 to smithi18136759-44 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:84: oid 44 updating version 0 to 12 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:84: oid 44 version 12 is already newer than 11 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:84: oid 44 version 12 is already newer than 10 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 12 (ObjNum 59 snap 7 seq_num 59) dirty exists 2024-09-17T16:01:41.920 INFO:tasks.rados.rados.0.smithi181.stdout:84: left oid 44 (ObjNum 59 snap 7 seq_num 59) 2024-09-17T16:01:42.011 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:42.011 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:42.011 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:42.009+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:42.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:01:42.051 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:42.051 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:42.073 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:42.074 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:42.107 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:42.107 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:42.112 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:42.110+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:42.131 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:42.132 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:42.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:01:42.212 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:42.210+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:42.288 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:42.288 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:42.293 INFO:tasks.rados.rados.0.smithi181.stdout:82: finishing copy_from to smithi18136759-6 2024-09-17T16:01:42.293 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-17T16:01:42.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:01:42.313 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:42.310+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:42.332 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:42.332 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:42.352 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:42.353 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:42.404 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:42.404 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:42.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:01:42.454 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:42.452+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:42.515 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:42.513+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:42.560 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:42.560 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:42.565 INFO:tasks.rados.rados.0.smithi181.stdout:81: finishing copy_from to smithi18136759-5 2024-09-17T16:01:42.565 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 10 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-17T16:01:42.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:01:42.616 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:42.614+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:42.689 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:42.689 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:42.689 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:42.689 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:42.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:01:42.717 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:42.715+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:42.806 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:42.806 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:42.807 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:42.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:01:42.818 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:42.816+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:42.896 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:81: done (4 left) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:82: done (3 left) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:83: done (2 left) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:84: done (1 left) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:85: done (0 left) 2024-09-17T16:01:42.899 INFO:tasks.rados.rados.0.smithi181.stdout:86: delete oid 4 current snap is 8 2024-09-17T16:01:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:86: done (0 left) 2024-09-17T16:01:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:87: rollback oid 5 current snap is 8 2024-09-17T16:01:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 5 to 5 2024-09-17T16:01:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 20 current snap is 8 2024-09-17T16:01:42.906 INFO:tasks.rados.rados.0.smithi181.stdout:88: seq_num 61 ranges {1245184=655360,1900544=376832} 2024-09-17T16:01:42.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-17T16:01:42.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:01:42.917 INFO:tasks.rados.rados.0.smithi181.stdout:88: writing smithi18136759-20 from 1245184 to 1900544 tid 1 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:88: writing smithi18136759-20 from 1900544 to 2277376 tid 2 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:89: setattr oid 4 current snap is 8 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:87: finishing rollback tid 0 to smithi18136759-5 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:87: finishing rollback tid 1 to smithi18136759-5 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:87: finishing rollback tid 2 to smithi18136759-5 2024-09-17T16:01:42.921 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 14 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-17T16:01:42.922 INFO:tasks.rados.rados.0.smithi181.stdout:87: done (2 left) 2024-09-17T16:01:42.922 INFO:tasks.rados.rados.0.smithi181.stdout:90: rollback oid 11 current snap is 8 2024-09-17T16:01:42.922 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 11 to 3 2024-09-17T16:01:42.922 INFO:tasks.rados.rados.0.smithi181.stdout:91: copy_from oid 31 from oid 38 current snap is 8 2024-09-17T16:01:42.922 INFO:tasks.rados.rados.0.smithi181.stdout:92: rollback oid 26 current snap is 8 2024-09-17T16:01:42.923 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 26 to 3 2024-09-17T16:01:42.923 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 3 current snap is 8 2024-09-17T16:01:42.923 INFO:tasks.rados.rados.0.smithi181.stdout:93: seq_num 62 ranges {49152=671744,720896=376832} 2024-09-17T16:01:42.927 INFO:tasks.rados.rados.0.smithi181.stdout:93: writing smithi18136759-3 from 49152 to 720896 tid 1 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=145, reqid=client.4560.0:174, at_version=38'9, trim_to=0'0, roll_forward_to=35'7))", 2024-09-17T16:01:42.928 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.869616+0000", 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0558548319999996, 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00091643599999999997, 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869616+0000", 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869616+0000", 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869616+0000", 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0599999999999999e-07 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869640+0000", 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3757e-05 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869674+0000", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4201e-05 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869685+0000", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0699e-05 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869733+0000", 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8353999999999998e-05 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.869741+0000", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7829999999999999e-06 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.870501+0000", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076035600000000001 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.870532+0000", 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1080000000000001e-05 2024-09-17T16:01:42.932 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=147, reqid=client.4560.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.874418+0000", 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0510523779999996, 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052883699999999999, 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874418+0000", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874418+0000", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874419+0000", 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0999999999999999e-07 2024-09-17T16:01:42.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874427+0000", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1939999999999998e-06 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874448+0000", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.071e-05 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874456+0000", 2024-09-17T16:01:42.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8790000000000003e-06 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874505+0000", 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9753999999999998e-05 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874512+0000", 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2179999999999996e-06 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874926+0000", 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041420600000000003 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.874947+0000", 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1365999999999998e-05 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/32 ECSubWrite(tid=11, reqid=client.4560.0:172, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:42.937 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.877693+0000", 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.04777728, 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013460060000000001, 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.877693+0000", 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.877693+0000", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.877694+0000", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9000000000000002e-07 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.877993+0000", 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029892599999999998 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.878027+0000", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3995000000000002e-05 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.878036+0000", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0170000000000002e-06 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.878085+0000", 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9264999999999997e-05 2024-09-17T16:01:42.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.878090+0000", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2469999999999999e-06 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.879015+0000", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000924605 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.879039+0000", 2024-09-17T16:01:42.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4561000000000001e-05 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 38/32 ECSubWrite(tid=12, reqid=client.4560.0:172, at_version=0'0, trim_to=0'0, roll_forward_to=38'8))", 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.880826+0000", 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0446449839999996, 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068566899999999997, 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.942 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880826+0000", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880826+0000", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880826+0000", 2024-09-17T16:01:42.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.06e-07 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880836+0000", 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3700000000000001e-06 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880864+0000", 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7849000000000001e-05 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880870+0000", 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.015e-06 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880925+0000", 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5226000000000003e-05 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.880935+0000", 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0242e-05 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881479+0000", 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054395099999999998 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881511+0000", 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2310000000000001e-05 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.946 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=148, reqid=client.4560.0:175, at_version=38'11, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.881050+0000", 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0444209329999996, 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001389515, 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881050+0000", 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881050+0000", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881051+0000", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0100000000000001e-06 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881443+0000", 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000391739 2024-09-17T16:01:42.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881479+0000", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6303999999999997e-05 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881487+0000", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.354e-06 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881531+0000", 2024-09-17T16:01:42.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3544000000000002e-05 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.881537+0000", 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2670000000000003e-06 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.882414+0000", 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000876817 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.882439+0000", 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.548e-05 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=149, reqid=client.4560.0:176, at_version=38'12, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.883577+0000", 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0418938090000003, 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088395300000000004, 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.951 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883577+0000", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883577+0000", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883578+0000", 2024-09-17T16:01:42.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0999999999999998e-07 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883762+0000", 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000184363 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883797+0000", 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4635999999999998e-05 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883807+0000", 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9830000000000008e-06 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883871+0000", 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4725000000000001e-05 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.883877+0000", 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1800000000000001e-06 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884437+0000", 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055928100000000001 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884461+0000", 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4074999999999999e-05 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.955 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.884662+0000", 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0408086010000002, 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058666499999999999, 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884662+0000", 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884662+0000", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884663+0000", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884678+0000", 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5671e-05 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884711+0000", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2246000000000003e-05 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884720+0000", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7720000000000006e-06 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884789+0000", 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8236000000000002e-05 2024-09-17T16:01:42.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.884795+0000", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1550000000000004e-06 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.885223+0000", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042806799999999998 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.885249+0000", 2024-09-17T16:01:42.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5876000000000001e-05 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.886280+0000", 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0391911770000002, 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064241400000000001, 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.960 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886280+0000", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886280+0000", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886280+0000", 2024-09-17T16:01:42.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3099999999999997e-07 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886297+0000", 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6895999999999999e-05 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886334+0000", 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6638999999999999e-05 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886344+0000", 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0020999999999999e-05 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886409+0000", 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4806000000000003e-05 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886415+0000", 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1229999999999997e-06 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886897+0000", 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000482712 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.886922+0000", 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4485999999999999e-05 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.964 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.887070+0000", 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0384010139999997, 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055100400000000001, 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887070+0000", 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887070+0000", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887070+0000", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5000000000000002e-07 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887097+0000", 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6577999999999999e-05 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4270999999999998e-05 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887141+0000", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7960000000000007e-06 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887178+0000", 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7030000000000003e-05 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887183+0000", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887603+0000", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041941400000000001 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.887621+0000", 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7776999999999999e-05 2024-09-17T16:01:42.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=154, reqid=client.4560.0:183, at_version=0'0, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:36.892253+0000", 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0332176100000003, 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051812799999999997, 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892253+0000", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892253+0000", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892254+0000", 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892266+0000", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2573e-05 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892301+0000", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4109999999999997e-05 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892309+0000", 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3899999999999993e-06 2024-09-17T16:01:42.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892376+0000", 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6765000000000002e-05 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892382+0000", 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0519999999999998e-06 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892751+0000", 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036956000000000001 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:36.892771+0000", 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9904000000000001e-05 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "pg_update_log_missing(3.8s1 epoch 39/32 rep_tid 155 entries 39'16 (0'0) error 3:19aefe3f:::smithi18136759-1:head by client.4560.0:187 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 38'14)", 2024-09-17T16:01:42.973 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.875864+0000", 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0496062449999997, 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057410999999999996, 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875864+0000", 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875864+0000", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875865+0000", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2800000000000001e-07 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875873+0000", 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6149999999999997e-06 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875916+0000", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2395000000000003e-05 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.875927+0000", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876006+0000", 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9609000000000003e-05 2024-09-17T16:01:42.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876439+0000", 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043238499999999997 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 39/32 ECSubWrite(tid=156, reqid=osd.11.0:156, at_version=39'18, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.876902+0000", 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.048568242, 2024-09-17T16:01:42.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092299299999999999, 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876902+0000", 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876902+0000", 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876903+0000", 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.75e-07 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876925+0000", 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1489e-05 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876959+0000", 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4480999999999997e-05 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.876967+0000", 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3839999999999997e-06 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877069+0000", 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101693 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877079+0000", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5249999999999994e-06 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877786+0000", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00070748600000000001 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877825+0000", 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9360000000000003e-05 2024-09-17T16:01:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 39/32 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=39'20, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.877908+0000", 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0475627210000003, 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065766800000000003, 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877908+0000", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877908+0000", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877908+0000", 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7599999999999997e-07 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877923+0000", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4644000000000001e-05 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877942+0000", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9340000000000001e-05 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.877948+0000", 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4659999999999998e-06 2024-09-17T16:01:42.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.878009+0000", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1248999999999996e-05 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.878016+0000", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6089999999999997e-06 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.878525+0000", 2024-09-17T16:01:42.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050970099999999999 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.878566+0000", 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0182999999999998e-05 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 39/32 ECSubWrite(tid=13, reqid=osd.15.0:13, at_version=39'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.880975+0000", 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0444953559999997, 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066072199999999998, 2024-09-17T16:01:42.986 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.880975+0000", 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.880975+0000", 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.880976+0000", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.880990+0000", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4810000000000001e-05 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881007+0000", 2024-09-17T16:01:42.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6368999999999999e-05 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881012+0000", 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6670000000000001e-06 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881054+0000", 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1984999999999999e-05 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881060+0000", 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.327e-06 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881604+0000", 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054409399999999996 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.881636+0000", 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2181000000000002e-05 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 39/32 ECSubWrite(tid=158, reqid=osd.11.0:157, at_version=0'0, trim_to=0'0, roll_forward_to=39'20))", 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.884236+0000", 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0412347860000004, 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000576199, 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884236+0000", 2024-09-17T16:01:42.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884236+0000", 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884236+0000", 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7199999999999999e-07 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884246+0000", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5060000000000001e-06 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884270+0000", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4114e-05 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884276+0000", 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7930000000000003e-06 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884326+0000", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0416e-05 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884333+0000", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1489999999999997e-06 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884781+0000", 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044761900000000001 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.884812+0000", 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1029999999999999e-05 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 39/32 ECSubWrite(tid=14, reqid=osd.15.0:13, at_version=0'0, trim_to=0'0, roll_forward_to=39'10))", 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.886669+0000", 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0388014630000004, 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072001900000000002, 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:42.995 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.886669+0000", 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.886669+0000", 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.886670+0000", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4899999999999995e-07 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.886678+0000", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2199999999999992e-06 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887005+0000", 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032738100000000002 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887011+0000", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.536e-06 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887053+0000", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1990000000000003e-05 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887059+0000", 2024-09-17T16:01:42.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2419999999999997e-06 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887363+0000", 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030376300000000003 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887389+0000", 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6237999999999998e-05 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:42.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 43/32 ECSubWrite(tid=30, reqid=client.4560.0:217, at_version=0'0, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.292651+0000", 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.63281968899999996, 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088639199999999995, 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292651+0000", 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292651+0000", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292652+0000", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.418e-06 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292667+0000", 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4124000000000001e-05 2024-09-17T16:01:43.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292716+0000", 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9026999999999997e-05 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292729+0000", 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.342e-05 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292817+0000", 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7973000000000006e-05 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.292829+0000", 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2462e-05 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.293470+0000", 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064070299999999995 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.293537+0000", 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7265000000000001e-05 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.004 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.006 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:42.952+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:43.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:93: writing smithi18136759-3 from 720896 to 1097728 tid 2 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:94: read oid 29 snap -1 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:94: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:95: copy_from oid 6 from oid 17 current snap is 8 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 1 to smithi18136759-20 2024-09-17T16:01:43.011 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 2 to smithi18136759-20 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing rollback tid 1 to smithi18136759-11 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:96: read oid 5 snap -1 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:96: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:97: read oid 30 snap -1 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:97: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:98: rollback oid 41 current snap is 8 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 41 to 7 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 42 current snap is 8 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:99: seq_num 63 ranges {802816=688128,1490944=376832} 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:99: writing smithi18136759-42 from 802816 to 1490944 tid 1 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:99: writing smithi18136759-42 from 1490944 to 1867776 tid 2 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:100: snap_create 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:90: finishing rollback tid 2 to smithi18136759-11 2024-09-17T16:01:43.012 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 11 v 10 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 9 (ObjNum 1988074736 snap 32637 seq_num 304) dirty exists 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 0 to smithi18136759-26 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 1 to smithi18136759-26 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 3 to smithi18136759-20 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:92: finishing rollback tid 2 to smithi18136759-26 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 8 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:93: finishing write tid 1 to smithi18136759-3 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:96: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 2878 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 33886 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:88: finishing write tid 4 to smithi18136759-20 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 20 updating version 0 to 25 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 20 version 25 is already newer than 23 2024-09-17T16:01:43.013 INFO:tasks.rados.rados.0.smithi181.stdout:88: oid 20 version 25 is already newer than 24 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 25 (ObjNum 61 snap 8 seq_num 61) dirty exists 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:88: left oid 20 (ObjNum 61 snap 8 seq_num 61) 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:98: finishing rollback tid 0 to smithi18136759-41 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:91: finishing copy_from racing read to smithi18136759-31 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:93: finishing write tid 2 to smithi18136759-3 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:98: finishing rollback tid 1 to smithi18136759-41 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:97: expect (ObjNum 29 snap 0 seq_num 29) 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 231465 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 176704 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:98: finishing rollback tid 2 to smithi18136759-41 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:93: finishing write tid 3 to smithi18136759-3 2024-09-17T16:01:43.014 INFO:tasks.rados.rados.0.smithi181.stdout:93: finishing write tid 4 to smithi18136759-3 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:93: oid 3 updating version 0 to 13 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:93: oid 3 version 13 is already newer than 11 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:93: oid 3 version 13 is already newer than 12 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 13 (ObjNum 62 snap 8 seq_num 62) dirty exists 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:93: left oid 3 (ObjNum 62 snap 8 seq_num 62) 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing copy_from racing read to smithi18136759-6 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:91: finishing copy_from to smithi18136759-31 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 1 to smithi18136759-42 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:95: finishing copy_from to smithi18136759-6 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 2 to smithi18136759-42 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 3 to smithi18136759-42 2024-09-17T16:01:43.015 INFO:tasks.rados.rados.0.smithi181.stdout:99: finishing write tid 4 to smithi18136759-42 2024-09-17T16:01:43.016 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 42 updating version 0 to 14 2024-09-17T16:01:43.016 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 42 version 14 is already newer than 13 2024-09-17T16:01:43.016 INFO:tasks.rados.rados.0.smithi181.stdout:99: oid 42 updating version 14 to 15 2024-09-17T16:01:43.016 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 15 (ObjNum 63 snap 8 seq_num 63) dirty exists 2024-09-17T16:01:43.016 INFO:tasks.rados.rados.0.smithi181.stdout:99: left oid 42 (ObjNum 63 snap 8 seq_num 63) 2024-09-17T16:01:43.019 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:43.018+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:43.049 INFO:tasks.rados.rados.0.smithi181.stdout:94: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-17T16:01:43.051 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 287761 2024-09-17T16:01:43.053 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:43.059 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 438622 2024-09-17T16:01:43.118 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.118 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:43.118 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:43.118 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.120 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:43.118+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:43.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:01:43.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:43.177 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:43.198 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:43.198 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:43.220 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:43.220 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:43.221 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:43.219+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:43.234 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.234 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:43.235 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:43.235 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:43.235 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.242 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:43.242 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:43.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:01:43.263 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:43.263 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:43.284 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:43.284 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:43.306 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:43.306 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:43.322 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:43.320+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:43.326 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:43.326 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:43.347 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:43.347 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:43.348 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:43.346+0000 7f0060eee640 0 -- 172.21.15.6:0/3547476141 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7f005c221690 msgr2=0x7f005c223a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:43.391 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=31, reqid=client.4560.0:225, at_version=44'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:43.392 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.48636794, 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011247220000000001, 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:43.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903503+0000", 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.23e-07 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903525+0000", 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2575999999999999e-05 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903562+0000", 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6705000000000002e-05 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903571+0000", 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.054e-06 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903639+0000", 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8058999999999993e-05 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903651+0000", 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2313000000000001e-05 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.904589+0000", 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093754699999999995 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.904627+0000", 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8345e-05 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.397 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=32, reqid=client.4560.0:225, at_version=0'0, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.905540+0000", 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.48433062799999999, 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056059999999999997, 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905540+0000", 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905540+0000", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905540+0000", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905551+0000", 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0973e-05 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905581+0000", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9456000000000001e-05 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905591+0000", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7189999999999996e-06 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905650+0000", 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9862999999999999e-05 2024-09-17T16:01:43.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.905660+0000", 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0850000000000003e-06 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.906076+0000", 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041631399999999999 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.906100+0000", 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4516e-05 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=33, reqid=client.4560.0:233, at_version=44'9, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.46713676599999998, 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136534, 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922735+0000", 2024-09-17T16:01:43.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1340000000000001e-06 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922924+0000", 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018932500000000001 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922965+0000", 2024-09-17T16:01:43.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0547000000000003e-05 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922974+0000", 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4080000000000004e-06 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923042+0000", 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7964000000000002e-05 2024-09-17T16:01:43.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923056+0000", 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4018e-05 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924052+0000", 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000996432 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924099+0000", 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6511999999999998e-05 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.46123598500000001, 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.408 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:01:43.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=35, reqid=client.4560.0:242, at_version=44'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.459532567, 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023531030000000001, 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930339+0000", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1909999999999999e-06 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931084+0000", 2024-09-17T16:01:43.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074480300000000004 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931122+0000", 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8195000000000003e-05 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931132+0000", 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931291+0000", 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015850099999999999 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931302+0000", 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1487e-05 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932657+0000", 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001355001 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932691+0000", 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3426000000000002e-05 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=36, reqid=client.4560.0:243, at_version=44'13, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.45813612300000001, 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098751700000000004, 2024-09-17T16:01:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931735+0000", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931752+0000", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6582000000000001e-05 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931783+0000", 2024-09-17T16:01:43.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1599e-05 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931792+0000", 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6079999999999996e-06 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931919+0000", 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012768799999999999 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931928+0000", 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5590000000000006e-06 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932695+0000", 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076658200000000003 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932722+0000", 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7197000000000001e-05 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=37, reqid=client.4560.0:236, at_version=44'15, trim_to=0'0, roll_forward_to=44'11))", 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.45378829700000001, 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015795060000000001, 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936083+0000", 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.279e-06 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936265+0000", 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018191799999999999 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936310+0000", 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5139000000000003e-05 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936321+0000", 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0631e-05 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936421+0000", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9679000000000006e-05 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936430+0000", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1379999999999996e-06 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937627+0000", 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011968930000000001 2024-09-17T16:01:43.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937662+0000", 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4829000000000001e-05 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=42, reqid=client.4560.0:236, at_version=0'0, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.945872+0000", 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.44399834199999999, 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052842700000000004, 2024-09-17T16:01:43.426 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945872+0000", 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945872+0000", 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945873+0000", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9800000000000002e-07 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945884+0000", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1342000000000001e-05 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945915+0000", 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0241000000000001e-05 2024-09-17T16:01:43.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945922+0000", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-06 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945975+0000", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2944e-05 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.945983+0000", 2024-09-17T16:01:43.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2609999999999993e-06 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.946364+0000", 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038048599999999998 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.946400+0000", 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6764999999999998e-05 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.430 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.431 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:01:43.433 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:43.430+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:43.532 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:43.530+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:43.539 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.539 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:43.540 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:43.540 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:01:43.636 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:43.631+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:43.695 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:43.695 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:43.695 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:01:43.737 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:43.735+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4296.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.040584845000001, 2024-09-17T16:01:43.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012209969999999999, 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768808+0000", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.79e-07 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768818+0000", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000006e-06 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768864+0000", 2024-09-17T16:01:43.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5797999999999998e-05 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768875+0000", 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1610000000000001e-05 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768958+0000", 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3187000000000002e-05 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.768978+0000", 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0001e-05 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.769987+0000", 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001008793 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:19.770029+0000", 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1938999999999999e-05 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.4296.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.041907780999999, 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001137285, 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.817 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767485+0000", 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767486+0000", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7999999999999995e-07 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767494+0000", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0830000000000004e-06 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767563+0000", 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9461999999999993e-05 2024-09-17T16:01:43.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767579+0000", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5659999999999999e-05 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767700+0000", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012102700000000001 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.767745+0000", 2024-09-17T16:01:43.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4975000000000002e-05 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.768579+0000", 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083372899999999996 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:20.768622+0000", 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3769000000000001e-05 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:7, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9554484330000008, 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001460104, 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853946+0000", 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7230000000000001e-06 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854002+0000", 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677999999999998e-05 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854062+0000", 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-05 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854075+0000", 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2747e-05 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854189+0000", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114376 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854203+0000", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4100000000000001e-05 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855346+0000", 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011423010000000001 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855405+0000", 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8949000000000001e-05 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=1, reqid=client.4560.0:8, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9535874690000004, 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011212939999999999, 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.826 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855805+0000", 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855806+0000", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5999999999999991e-07 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855829+0000", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2616000000000002e-05 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855889+0000", 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0028000000000002e-05 2024-09-17T16:01:43.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855901+0000", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1732999999999999e-05 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855989+0000", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8177999999999995e-05 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855999+0000", 2024-09-17T16:01:43.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6870000000000006e-06 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856888+0000", 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00088945699999999996 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856927+0000", 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8634999999999999e-05 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9530657950000005, 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001864189, 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856328+0000", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3570000000000001e-06 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856466+0000", 2024-09-17T16:01:43.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137415 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856516+0000", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0541999999999998e-05 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856527+0000", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:01:43.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4474999999999994e-05 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856611+0000", 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0720000000000006e-06 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858143+0000", 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015325499999999999 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858191+0000", 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000003e-05 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9524443569999992, 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.835 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:01:43.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:01:43.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:01:43.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=3, reqid=client.4560.0:11, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9505184670000002, 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00096021099999999999, 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.842 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858874+0000", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858875+0000", 2024-09-17T16:01:43.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0359999999999999e-06 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858897+0000", 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1219000000000001e-05 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858943+0000", 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6718000000000002e-05 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858952+0000", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9449999999999999e-06 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859019+0000", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6594999999999995e-05 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859029+0000", 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0327e-05 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859792+0000", 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000762655 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859835+0000", 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2716000000000002e-05 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.846 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9494818649999992, 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:01:43.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:01:43.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9480567460000007, 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087109700000000004, 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861336+0000", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861337+0000", 2024-09-17T16:01:43.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861358+0000", 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1438e-05 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861402+0000", 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3157e-05 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861414+0000", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2638e-05 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861509+0000", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4903999999999998e-05 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861524+0000", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4313e-05 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.862172+0000", 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064898699999999996 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.862207+0000", 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4761000000000001e-05 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:43.855 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.945906742, 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001910307, 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:43.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863487+0000", 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863535+0000", 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7713999999999998e-05 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863653+0000", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117757 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863662+0000", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9779999999999994e-06 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863800+0000", 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013886599999999999 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863813+0000", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.295e-05 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865357+0000", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001543464 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865396+0000", 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9629000000000003e-05 2024-09-17T16:01:43.859 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9458515349999992, 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.860 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:01:43.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9444175189999999, 2024-09-17T16:01:43.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093210899999999998, 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864975+0000", 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864976+0000", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9400000000000004e-07 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864995+0000", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8513999999999999e-05 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865053+0000", 2024-09-17T16:01:43.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8217999999999997e-05 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0042e-05 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865137+0000", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4350000000000005e-05 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865148+0000", 2024-09-17T16:01:43.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0400999999999999e-05 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865868+0000", 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071984700000000002 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865907+0000", 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9943e-05 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=7, reqid=client.4560.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9411805290000004, 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092475100000000002, 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868212+0000", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868214+0000", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.192e-06 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868228+0000", 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4003e-05 2024-09-17T16:01:43.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868272+0000", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4536999999999997e-05 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868283+0000", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0781999999999999e-05 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868373+0000", 2024-09-17T16:01:43.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0136000000000006e-05 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868384+0000", 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0808000000000001e-05 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.869097+0000", 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071339000000000005 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.869137+0000", 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9903000000000001e-05 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=2, reqid=client.4560.0:61, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9247647109999999, 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018138760000000001, 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.873 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:43.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9e-08 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884974+0000", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034541900000000002 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885013+0000", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000002e-05 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885023+0000", 2024-09-17T16:01:43.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885103+0000", 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9871999999999997e-05 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885112+0000", 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4769999999999992e-06 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886400+0000", 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012878869999999999 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886442+0000", 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2202e-05 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=11, reqid=client.4560.0:91, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:43.877 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9016801579999996, 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001672984, 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:43.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907717+0000", 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6290000000000001e-06 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907950+0000", 2024-09-17T16:01:43.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023220000000000001 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907988+0000", 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8900999999999999e-05 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907997+0000", 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908086+0000", 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908108+0000", 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2447e-05 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909346+0000", 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001237782 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909386+0000", 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9554000000000001e-05 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=12, reqid=client.4560.0:92, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.9005765449999998, 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087827399999999996, 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.882 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908816+0000", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908817+0000", 2024-09-17T16:01:43.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.92e-07 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908843+0000", 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6046000000000001e-05 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908884+0000", 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1056000000000001e-05 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908894+0000", 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6330000000000008e-06 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908965+0000", 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0817999999999999e-05 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908975+0000", 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0587e-05 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909659+0000", 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068393700000000002 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909695+0000", 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5404999999999997e-05 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.886 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=13, reqid=client.4560.0:94, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.8981625659999999, 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012298319999999999, 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911231+0000", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1200000000000002e-07 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911496+0000", 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026446999999999999 2024-09-17T16:01:43.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911523+0000", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7665e-05 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911529+0000", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1530000000000002e-06 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911584+0000", 2024-09-17T16:01:43.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4611000000000003e-05 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911592+0000", 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912421+0000", 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082916399999999996 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912460+0000", 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9047000000000001e-05 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=17, reqid=client.4560.0:124, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.8607738299999994, 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016306890000000001, 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.891 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948620+0000", 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.238e-06 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949131+0000", 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051077299999999996 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949166+0000", 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4619999999999997e-05 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949174+0000", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2400000000000007e-06 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949224+0000", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0534000000000001e-05 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949231+0000", 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.229e-06 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950221+0000", 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098999599999999993 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950250+0000", 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9059000000000001e-05 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.92169071, 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:01:43.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:01:43.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:01:43.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.9171387500000003, 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:01:43.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:01:43.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:01:43.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:43.904 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:01:43.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:01:43.907 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:43.903+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:88: done (12 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:89: done (11 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:90: done (10 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:91: done (9 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:92: done (8 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:93: done (7 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:94: done (6 left) 2024-09-17T16:01:43.908 INFO:tasks.rados.rados.0.smithi181.stdout:95: done (5 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:96: done (4 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:97: done (3 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:98: done (2 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:99: done (1 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:100: done (0 left) 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 48 current snap is 9 2024-09-17T16:01:43.909 INFO:tasks.rados.rados.0.smithi181.stdout:101: seq_num 64 ranges {802816=704512,1507328=376832} 2024-09-17T16:01:43.915 INFO:tasks.rados.rados.0.smithi181.stdout:101: writing smithi18136759-48 from 802816 to 1507328 tid 1 2024-09-17T16:01:43.918 INFO:tasks.rados.rados.0.smithi181.stdout:101: writing smithi18136759-48 from 1507328 to 1884160 tid 2 2024-09-17T16:01:43.918 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 23 current snap is 9 2024-09-17T16:01:43.918 INFO:tasks.rados.rados.0.smithi181.stdout:102: seq_num 65 ranges {376832=720896,1097728=376832} 2024-09-17T16:01:43.924 INFO:tasks.rados.rados.0.smithi181.stdout:102: writing smithi18136759-23 from 376832 to 1097728 tid 1 2024-09-17T16:01:43.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-17T16:01:43.926 INFO:tasks.rados.rados.0.smithi181.stdout:102: writing smithi18136759-23 from 1097728 to 1474560 tid 2 2024-09-17T16:01:43.926 INFO:tasks.rados.rados.0.smithi181.stdout:103: rollback oid 26 current snap is 9 2024-09-17T16:01:43.927 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 26 to 6 2024-09-17T16:01:43.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: finishing write tid 1 to smithi18136759-48 2024-09-17T16:01:43.927 INFO:tasks.rados.rados.0.smithi181.stdout:104: delete oid 41 current snap is 9 2024-09-17T16:01:43.931 INFO:tasks.rados.rados.0.smithi181.stdout:103: finishing rollback tid 0 to smithi18136759-26 2024-09-17T16:01:43.931 INFO:tasks.rados.rados.0.smithi181.stdout:101: finishing write tid 2 to smithi18136759-48 2024-09-17T16:01:43.933 INFO:tasks.rados.rados.0.smithi181.stdout:101: finishing write tid 3 to smithi18136759-48 2024-09-17T16:01:43.933 INFO:tasks.rados.rados.0.smithi181.stdout:103: finishing rollback tid 1 to smithi18136759-26 2024-09-17T16:01:43.934 INFO:tasks.rados.rados.0.smithi181.stdout:104: done (3 left) 2024-09-17T16:01:43.934 INFO:tasks.rados.rados.0.smithi181.stdout:105: snap_remove snap 1 2024-09-17T16:01:44.007 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:44.004+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:44.054 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:44.054 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:44.054 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:44.054 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:44.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:01:44.108 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:44.106+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:44.169 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:44.169 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:44.169 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:44.169 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:44.170 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:44.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:01:44.210 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:44.207+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:44.310 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:44.308+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:5 3.bs0 3:d4004f6b:::smithi18136759-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:44.327 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.47200134, 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010865848000000001, 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2099999999999997e-07 2024-09-17T16:01:44.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852349+0000", 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3249999999999999e-06 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852356+0000", 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6040000000000004e-06 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852361+0000", 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.437e-06 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853722+0000", 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136107 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854082+0000", 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036069599999999998 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854755+0000", 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067232100000000003 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856366+0000", 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016111039999999999 2024-09-17T16:01:44.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863131+0000", 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067652729999999996 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863211+0000", 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9497000000000001e-05 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.471979115, 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:44.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.338 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:19 3.1s0 3:8347559a:::smithi18136759-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.471936725999999, 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010852832999999999, 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 19 2024-09-17T16:01:44.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852410+0000", 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852655+0000", 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024543600000000003 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852672+0000", 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6328e-05 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852682+0000", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0365e-05 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852763+0000", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0810999999999996e-05 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852923+0000", 2024-09-17T16:01:44.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016075700000000001 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853852+0000", 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092845700000000005 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856526+0000", 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026744960000000002 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863195+0000", 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066689100000000001 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863262+0000", 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6847000000000006e-05 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:44.344 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.471630156, 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:01:44.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:01:44.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.471590024999999, 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.350 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:44.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:01:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:46 3.6s0 3:6d55c691:::smithi18136759-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.460099862, 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010233671, 2024-09-17T16:01:44.355 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 46 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864248+0000", 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7400000000000001e-06 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865054+0000", 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000806429 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865069+0000", 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4673e-05 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865078+0000", 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6880000000000005e-06 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865530+0000", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000452724 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865698+0000", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016729700000000001 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.866696+0000", 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099818099999999989 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868021+0000", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013246709999999999 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874396+0000", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063754190000000002 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874480+0000", 2024-09-17T16:01:44.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3849000000000003e-05 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.458912731, 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:44.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:01:44.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:01:44.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.366 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.458639489999999, 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:01:44.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:01:44.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:01:44.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.455773708000001, 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.372 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:01:44.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.455721780999999, 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.378 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:01:44.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:66 3.6s0 3:60a7d8b5:::smithi18136759-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.443781785000001, 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010929104, 2024-09-17T16:01:44.383 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880565+0000", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880568+0000", 2024-09-17T16:01:44.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1939999999999998e-06 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880578+0000", 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-06 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880598+0000", 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9964999999999999e-05 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882449+0000", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018505730000000001 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882543+0000", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4135999999999995e-05 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890649+0000", 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081062559999999992 2024-09-17T16:01:44.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890748+0000", 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9234000000000006e-05 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891493+0000", 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074507600000000001 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.435181015, 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.389 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:01:44.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:01:44.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.435097840999999, 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:01:44.394 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:01:44.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:01:44.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:44.399 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.420655074000001, 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:01:44.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.420615384, 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:01:44.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:98 3.7s0 3:ec861466:::smithi18136759-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.410792282999999, 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099957889999999997, 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 98 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.411 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913555+0000", 2024-09-17T16:01:44.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4700000000000001e-07 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913560+0000", 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9969999999999998e-06 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913568+0000", 2024-09-17T16:01:44.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4740000000000005e-06 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913574+0000", 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7309999999999998e-06 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.914995+0000", 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001421381 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915104+0000", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010908000000000001 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915686+0000", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058229899999999999 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916507+0000", 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082023600000000001 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923452+0000", 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069455970000000004 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923550+0000", 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7246999999999999e-05 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:99 3.7s0 3:ec861466:::smithi18136759-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.410753682999999, 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011511970999999999, 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 99 2024-09-17T16:01:44.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913593+0000", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913596+0000", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2140000000000001e-06 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913605+0000", 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:01:44.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913612+0000", 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915969+0000", 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023571690000000001 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916049+0000", 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9777000000000005e-05 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924068+0000", 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080191269999999992 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924152+0000", 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4005999999999995e-05 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.925104+0000", 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095257899999999995 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:122 3.bs0 3:dee30a9d:::smithi18136759-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.381183612999999, 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011179774, 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 122 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943163+0000", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2200000000000004e-07 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943166+0000", 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:01:44.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943176+0000", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.166e-06 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943181+0000", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.502e-06 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944736+0000", 2024-09-17T16:01:44.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015548140000000001 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944814+0000", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7945e-05 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945207+0000", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039347299999999998 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945618+0000", 2024-09-17T16:01:44.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041055900000000001 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954241+0000", 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.008622691 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954342+0000", 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010164 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:123 3.bs0 3:dee30a9d:::smithi18136759-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.381155297999999, 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012212639000000001, 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 123 2024-09-17T16:01:44.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943194+0000", 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.565e-06 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943202+0000", 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0050000000000004e-06 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943207+0000", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3260000000000003e-06 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945420+0000", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022128310000000002 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945469+0000", 2024-09-17T16:01:44.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8897999999999998e-05 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954714+0000", 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092452019999999992 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954800+0000", 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5541e-05 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.955403+0000", 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060376199999999998 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.3853731439999999, 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:44.433 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:44.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:01:44.436 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:44.437 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:44.437 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:44.438 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:01:44.439 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:44.436+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:44.484 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:44.506 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:44.507 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:44.538 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:44.536+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:44.556 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:44.556 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:44.556 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:44.556 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:44.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:01:44.639 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:44.636+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:44.668 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:44.669 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:44.669 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:44.669 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:44.669 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:44.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:01:44.761 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:44.758+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:44.821 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:44.821 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:44.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:44.821 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:44.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:40 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'22, mlcod=25'19)", 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.895247994000002, 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069417209999999997, 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923346+0000", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923347+0000", 2024-09-17T16:01:44.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923352+0000", 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7929999999999997e-06 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923384+0000", 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2196000000000001e-05 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923394+0000", 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0369e-05 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923437+0000", 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3046000000000001e-05 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923451+0000", 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3556e-05 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930260+0000", 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068091169999999999 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.930288+0000", 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7937e-05 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.826 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:41 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'19)", 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.894709821999999, 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077219050000000003, 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923885+0000", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923947+0000", 2024-09-17T16:01:44.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1790000000000003e-05 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923974+0000", 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7368999999999998e-05 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923980+0000", 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924022+0000", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924037+0000", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4127e-05 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931579+0000", 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075422340000000001 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931606+0000", 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7307000000000001e-05 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:42 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'24, mlcod=25'19)", 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.894186992000002, 2024-09-17T16:01:44.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072286889999999999, 2024-09-17T16:01:44.832 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.832 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.832 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924408+0000", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0699999999999997e-07 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924412+0000", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924432+0000", 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9870000000000001e-05 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924436+0000", 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6e-06 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924471+0000", 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4527999999999999e-05 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924487+0000", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6303999999999998e-05 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931609+0000", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071217429999999998 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931636+0000", 2024-09-17T16:01:44.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6911e-05 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:43 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'25, mlcod=25'19)", 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.893615898, 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066863529999999999, 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.837 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924978+0000", 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924979+0000", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5600000000000005e-07 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924993+0000", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4431e-05 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925018+0000", 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5038999999999999e-05 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925023+0000", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6229999999999996e-06 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925060+0000", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6758999999999998e-05 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.925072+0000", 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.216e-05 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931638+0000", 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065666329999999997 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931664+0000", 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6052000000000001e-05 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.841 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:44 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'26, mlcod=25'20)", 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.892311924000001, 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054079590000000004, 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926282+0000", 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926283+0000", 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926321+0000", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7805000000000001e-05 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926348+0000", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7566e-05 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926354+0000", 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2290000000000002e-06 2024-09-17T16:01:44.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926396+0000", 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2391999999999997e-05 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.926409+0000", 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3298999999999999e-05 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931667+0000", 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052574520000000001 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931690+0000", 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3391999999999999e-05 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:48 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'30, mlcod=25'20)", 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:44.846 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.889712186000001, 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055739780000000003, 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928882+0000", 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6400000000000002e-07 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928898+0000", 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6059999999999999e-05 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928923+0000", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4961999999999999e-05 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928928+0000", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.921e-06 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928968+0000", 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9416999999999999e-05 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.928981+0000", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3548e-05 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934429+0000", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054479539999999996 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.934456+0000", 2024-09-17T16:01:44.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6551999999999999e-05 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:82 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'60, mlcod=25'58)", 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.775879442000001, 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0064218920000000002, 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042715+0000", 2024-09-17T16:01:44.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042718+0000", 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0879999999999999e-06 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042736+0000", 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7912999999999999e-05 2024-09-17T16:01:44.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042742+0000", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9669999999999998e-06 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042787+0000", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4610999999999997e-05 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.042801+0000", 2024-09-17T16:01:44.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.437e-05 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.049102+0000", 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063004239999999998 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.049136+0000", 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4958e-05 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:98 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'72, mlcod=25'66)", 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.678147791000001, 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005965237, 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140446+0000", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140447+0000", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3e-07 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140454+0000", 2024-09-17T16:01:44.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3030000000000002e-06 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140481+0000", 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6506e-05 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140487+0000", 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9619999999999998e-06 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140519+0000", 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2567999999999997e-05 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140531+0000", 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1321999999999999e-05 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146386+0000", 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005855206 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146411+0000", 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5639999999999998e-05 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:99 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'73, mlcod=25'66)", 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 26.677691906, 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055288220000000001, 2024-09-17T16:01:44.860 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140902+0000", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140903+0000", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5200000000000002e-07 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140968+0000", 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5388999999999993e-05 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140986+0000", 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7946999999999999e-05 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.140992+0000", 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8200000000000002e-06 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141026+0000", 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4745e-05 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.141040+0000", 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3997e-05 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146413+0000", 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053725379999999996 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:18.146431+0000", 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7934000000000001e-05 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(osd.14.22:1 3.es0 3:7408d595:::smithi18136759-48:head [assert-version v7,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.9391239569999996, 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059963730000000002, 2024-09-17T16:01:44.865 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879470+0000", 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879471+0000", 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3170000000000001e-06 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879476+0000", 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5839999999999996e-06 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879515+0000", 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8578000000000002e-05 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879519+0000", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5059999999999997e-06 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879568+0000", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8609000000000002e-05 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.879647+0000", 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9402999999999999e-05 2024-09-17T16:01:44.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.885466+0000", 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058193760000000002 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.9113712980000002, 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:01:44.870 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:01:44.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:44.874 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.9099013870000001, 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:01:44.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.903548672, 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.879 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:01:44.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.90156608500000002, 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.886 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:01:44.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:01:44.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.892 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.90114021700000002, 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.893 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:01:44.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:01:44.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:01:44.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.90111204700000003, 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:44.899 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:44.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:01:44.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.4s1 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.88994889200000005, 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077955969999999996, 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.905 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928646+0000", 2024-09-17T16:01:44.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928660+0000", 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4535000000000001e-05 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928684+0000", 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3938e-05 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928692+0000", 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928739+0000", 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7574999999999998e-05 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928746+0000", 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936413+0000", 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076669930000000004 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936441+0000", 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7413999999999999e-05 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.909 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.88831119400000003, 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:01:44.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:44.914 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.88799156700000004, 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:01:44.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:01:44.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.88789975300000001, 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:44.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:44.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:44.923 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:44.925 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:01:44.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:01:44.926 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:44.922+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:105: done (3 left) 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:106: snap_create 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: finishing write tid 4 to smithi18136759-48 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: oid 48 updating version 0 to 16 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: oid 48 version 16 is already newer than 14 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: oid 48 version 16 is already newer than 15 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 16 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:101: left oid 48 (ObjNum 64 snap 9 seq_num 64) 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:103: finishing rollback tid 2 to smithi18136759-26 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T16:01:44.927 INFO:tasks.rados.rados.0.smithi181.stdout:102: finishing write tid 1 to smithi18136759-23 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: finishing write tid 2 to smithi18136759-23 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: finishing write tid 3 to smithi18136759-23 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: finishing write tid 4 to smithi18136759-23 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: oid 23 updating version 0 to 17 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: oid 23 updating version 17 to 18 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: oid 23 version 18 is already newer than 16 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 23 v 18 (ObjNum 65 snap 9 seq_num 65) dirty exists 2024-09-17T16:01:44.928 INFO:tasks.rados.rados.0.smithi181.stdout:102: left oid 23 (ObjNum 65 snap 9 seq_num 65) 2024-09-17T16:01:44.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-17T16:01:45.026 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:45.024+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:45.070 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:45.070 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:45.070 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.079 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:01:45.126 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:45.124+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:45.183 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.183 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:45.183 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:45.183 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:45.183 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.192 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:01:45.235 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:45.232+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:45.297 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.297 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:45.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:45.297 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:45.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=147, reqid=client.4560.0:174, at_version=0'0, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.4196803389999992, 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022903149999999998, 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:45.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874463+0000", 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874464+0000", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9599999999999998e-07 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874479+0000", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4235e-05 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874516+0000", 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6990000000000003e-05 2024-09-17T16:01:45.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874524+0000", 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6370000000000005e-06 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874587+0000", 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2878000000000003e-05 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.874597+0000", 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0088000000000001e-05 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.876717+0000", 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002119606 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.876754+0000", 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6984999999999999e-05 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:45.303 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.4094592040000009, 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038886110000000002, 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884707+0000", 2024-09-17T16:01:45.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2036999999999999e-05 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884745+0000", 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7843999999999999e-05 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884756+0000", 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1240000000000001e-05 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884826+0000", 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9873e-05 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884836+0000", 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0053e-05 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888486+0000", 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036497270000000002 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888573+0000", 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7068000000000001e-05 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:45.308 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.4078494090000007, 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:01:45.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:45.313 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.4070381659999995, 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044954000000000001, 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:01:45.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4892000000000002e-05 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887167+0000", 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5617999999999997e-05 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887176+0000", 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6379999999999993e-06 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887237+0000", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1760000000000005e-05 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887246+0000", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891572+0000", 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004325626 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891601+0000", 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9105e-05 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.3747760119999999, 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027013699999999998, 2024-09-17T16:01:45.318 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.58e-07 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919591+0000", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022241000000000001 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919654+0000", 2024-09-17T16:01:45.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3744999999999994e-05 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919663+0000", 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7909999999999993e-06 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919742+0000", 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8751999999999994e-05 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919753+0000", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.111e-05 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922038+0000", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002285418 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922069+0000", 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0685999999999997e-05 2024-09-17T16:01:45.322 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 45/32 ECSubWrite(tid=27, reqid=client.4560.0:266, at_version=45'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.371732924, 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002821306, 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922412+0000", 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0800000000000004e-07 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922427+0000", 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5136999999999999e-05 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922462+0000", 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5172999999999997e-05 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922471+0000", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922539+0000", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8458000000000001e-05 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922551+0000", 2024-09-17T16:01:45.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2106999999999999e-05 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925193+0000", 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002641539 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925232+0000", 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9298000000000002e-05 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 45/32 ECSubWrite(tid=66, reqid=client.4560.0:275, at_version=0'0, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.360840848, 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034407219999999998, 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933311+0000", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0229999999999992e-06 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933334+0000", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2843000000000001e-05 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933340+0000", 2024-09-17T16:01:45.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.524e-06 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933385+0000", 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5037000000000003e-05 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933392+0000", 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936713+0000", 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033208700000000001 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936744+0000", 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0935000000000001e-05 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.332 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 46/32 ECSubWrite(tid=68, reqid=osd.13.0:68, at_version=46'12, trim_to=0'0, roll_forward_to=42'10))", 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.917380+0000", 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.37676365099999998, 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022416279999999998, 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917380+0000", 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917380+0000", 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917381+0000", 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7100000000000002e-07 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917399+0000", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8456000000000001e-05 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917470+0000", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0672000000000004e-05 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917480+0000", 2024-09-17T16:01:45.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.027e-05 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917780+0000", 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029999200000000001 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917789+0000", 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.67e-06 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919581+0000", 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017919870000000001 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919622+0000", 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1010000000000002e-05 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.337 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 46/32 ECSubWrite(tid=30, reqid=osd.2.0:30, at_version=46'18, trim_to=0'0, roll_forward_to=45'16))", 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.917414+0000", 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.37673015700000001, 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022444399999999999, 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917414+0000", 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917414+0000", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917414+0000", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917429+0000", 2024-09-17T16:01:45.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5489e-05 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917756+0000", 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032700200000000001 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917764+0000", 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.391e-06 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917822+0000", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8399000000000001e-05 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.917830+0000", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9130000000000003e-06 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919631+0000", 2024-09-17T16:01:45.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001801172 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919658+0000", 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6879000000000001e-05 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=183, reqid=osd.11.0:183, at_version=46'27, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.918849+0000", 2024-09-17T16:01:45.342 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.375294868, 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022168470000000001, 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918849+0000", 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918849+0000", 2024-09-17T16:01:45.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918849+0000", 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7599999999999999e-07 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918878+0000", 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8824999999999999e-05 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918917+0000", 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8747e-05 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918925+0000", 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918998+0000", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2982e-05 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919007+0000", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9460000000000004e-06 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921027+0000", 2024-09-17T16:01:45.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00202031 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921066+0000", 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8442999999999998e-05 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:45.347 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.37283524099999998, 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024618610000000001, 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7199999999999999e-07 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921323+0000", 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3711e-05 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921350+0000", 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7110999999999998e-05 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921355+0000", 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5060000000000003e-06 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921406+0000", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0337000000000003e-05 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921415+0000", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6350000000000001e-06 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923739+0000", 2024-09-17T16:01:45.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023234620000000001 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923770+0000", 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1526999999999998e-05 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 46/32 ECSubWrite(tid=69, reqid=osd.13.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:45.352 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.370827613, 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030339099999999999, 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923317+0000", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923325+0000", 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4279999999999996e-06 2024-09-17T16:01:45.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923347+0000", 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1980999999999998e-05 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923353+0000", 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3900000000000001e-06 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923391+0000", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8013999999999999e-05 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923396+0000", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1529999999999996e-06 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926319+0000", 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002923602 2024-09-17T16:01:45.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926350+0000", 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0716000000000002e-05 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.37004453300000001, 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054480279999999997, 2024-09-17T16:01:45.357 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924122+0000", 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2844999999999999e-05 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924159+0000", 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6534e-05 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924166+0000", 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.742e-06 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924210+0000", 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4641999999999997e-05 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924220+0000", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7580000000000004e-06 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929512+0000", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005292265 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929547+0000", 2024-09-17T16:01:45.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5046999999999998e-05 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=51, reqid=osd.12.0:51, at_version=46'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.369937395, 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053997719999999997, 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.362 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924207+0000", 2024-09-17T16:01:45.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924223+0000", 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6835000000000001e-05 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924252+0000", 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8285000000000001e-05 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924259+0000", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924320+0000", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1709000000000002e-05 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924329+0000", 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8799999999999997e-06 2024-09-17T16:01:45.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929582+0000", 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052531169999999999 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929606+0000", 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3913e-05 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.36911515499999997, 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:45.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:01:45.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 46/32 ECSubWrite(tid=70, reqid=osd.13.0:67, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:45.371 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.36878776899999999, 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042172039999999996, 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925357+0000", 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925366+0000", 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-09-17T16:01:45.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925396+0000", 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9964e-05 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925404+0000", 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6429999999999995e-06 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925470+0000", 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6817999999999996e-05 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925480+0000", 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929543+0000", 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040623580000000003 2024-09-17T16:01:45.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929573+0000", 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0682000000000002e-05 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:45.376 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.36675619300000001, 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:01:45.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.364355551, 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052541769999999996, 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.381 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929789+0000", 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929800+0000", 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1341e-05 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929829+0000", 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8642000000000001e-05 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929837+0000", 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929901+0000", 2024-09-17T16:01:45.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4067999999999997e-05 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929907+0000", 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.609e-06 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934990+0000", 2024-09-17T16:01:45.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050827579999999997 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935042+0000", 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2676999999999997e-05 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=39, reqid=osd.0.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.36159539699999999, 2024-09-17T16:01:45.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050006720000000003, 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:45.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932549+0000", 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932561+0000", 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1674e-05 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932581+0000", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0024999999999999e-05 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932590+0000", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7780000000000002e-06 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932642+0000", 2024-09-17T16:01:45.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1753999999999999e-05 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932651+0000", 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2310000000000002e-06 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937518+0000", 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048669969999999996 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937549+0000", 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0651999999999997e-05 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=54, reqid=osd.12.0:53, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.35612606200000002, 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028466110000000002, 2024-09-17T16:01:45.391 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:45.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938026+0000", 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9209999999999992e-06 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938055+0000", 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8252999999999999e-05 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938061+0000", 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0249999999999999e-06 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938102+0000", 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1875000000000001e-05 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938109+0000", 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4200000000000004e-06 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940826+0000", 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002716754 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940864+0000", 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8736999999999999e-05 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.396 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.397 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:45.395+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:45.398 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:01:45.498 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:45.496+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:45.541 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.542 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:45.542 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:45.542 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.550 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:01:45.599 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:45.597+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:45.654 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.654 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:45.654 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:45.654 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:45.654 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:01:45.700 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:45.698+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.90911006, 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.767 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:01:45.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:01:45.772 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.772 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.772 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.908209186000001, 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:45.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:01:45.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.904119358999999, 2024-09-17T16:01:45.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:01:45.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:01:45.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.902780912000001, 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055595449999999999, 2024-09-17T16:01:45.782 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:45.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861313+0000", 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9800000000000002e-07 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861333+0000", 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0276999999999999e-05 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861372+0000", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9365e-05 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861382+0000", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3349999999999996e-06 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861456+0000", 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3972000000000003e-05 2024-09-17T16:01:45.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861465+0000", 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7480000000000005e-06 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866841+0000", 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053751720000000001 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866871+0000", 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0678e-05 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.901815812000001, 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045470969999999999, 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.789 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862278+0000", 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5199999999999995e-07 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862373+0000", 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5013999999999995e-05 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862431+0000", 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862440+0000", 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6339999999999996e-06 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862513+0000", 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3028999999999995e-05 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862522+0000", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866779+0000", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042571140000000002 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866824+0000", 2024-09-17T16:01:45.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4511000000000001e-05 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.901712398000001, 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.794 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:01:45.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.798 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.89988702, 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:45.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:01:45.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:45.803 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.899738944999999, 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:45.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.807 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.899089061, 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:01:45.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 35/32 ECSubWrite(tid=6, reqid=client.4560.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.896531522, 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041413530000000004, 2024-09-17T16:01:45.813 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867561+0000", 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867562+0000", 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6599999999999994e-07 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867576+0000", 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4229999999999999e-05 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867702+0000", 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012607699999999999 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867712+0000", 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867779+0000", 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7217000000000004e-05 2024-09-17T16:01:45.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867789+0000", 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6700000000000006e-06 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871665+0000", 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003875898 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871702+0000", 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7444000000000003e-05 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.896511697999999, 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.818 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:45.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.822 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.896020526000001, 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:01:45.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:01:45.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.827 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.895895028, 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0035317220000000002, 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868198+0000", 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868208+0000", 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6989999999999998e-06 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868243+0000", 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5157000000000002e-05 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868250+0000", 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1489999999999997e-06 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868323+0000", 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2859000000000002e-05 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868334+0000", 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0601000000000001e-05 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871706+0000", 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033717389999999999 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871729+0000", 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3648000000000001e-05 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.832 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=6, reqid=client.4560.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.895283675, 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048136530000000002, 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868810+0000", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9099999999999991e-07 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868951+0000", 2024-09-17T16:01:45.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014129799999999999 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868987+0000", 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5556000000000003e-05 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868996+0000", 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6549999999999999e-06 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869062+0000", 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5714000000000001e-05 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869072+0000", 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0273e-05 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873569+0000", 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044970189999999997 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873623+0000", 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3146999999999998e-05 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.918432+0000", 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.84566017100000002, 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025127550000000002, 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.837 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918432+0000", 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918432+0000", 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.91e-07 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918456+0000", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2926999999999999e-05 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918504+0000", 2024-09-17T16:01:45.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8093000000000002e-05 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0037999999999999e-05 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918600+0000", 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6114999999999998e-05 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.918610+0000", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7489999999999993e-06 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920903+0000", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002293064 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920945+0000", 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2478000000000002e-05 2024-09-17T16:01:45.841 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.923776+0000", 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.84031637000000003, 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027692630000000001, 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923776+0000", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923776+0000", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923776+0000", 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.11e-07 2024-09-17T16:01:45.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923799+0000", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2167e-05 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923835+0000", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6374000000000001e-05 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923843+0000", 2024-09-17T16:01:45.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5549999999999997e-06 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923915+0000", 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2251999999999999e-05 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923925+0000", 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9490000000000008e-06 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926512+0000", 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002587457 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926545+0000", 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3297999999999999e-05 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.846 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.83902159799999998, 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043002459999999998, 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925072+0000", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925120+0000", 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8337999999999997e-05 2024-09-17T16:01:45.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925170+0000", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9965e-05 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925178+0000", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2220000000000002e-06 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925233+0000", 2024-09-17T16:01:45.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4378e-05 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925239+0000", 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2090000000000002e-06 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929339+0000", 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041006410000000004 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929371+0000", 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1798000000000003e-05 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.83896940799999997, 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042998380000000003, 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.851 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925124+0000", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8299999999999998e-07 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925141+0000", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7156e-05 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925203+0000", 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1990999999999997e-05 2024-09-17T16:01:45.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925208+0000", 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7230000000000001e-06 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925267+0000", 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8118000000000002e-05 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925275+0000", 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9299999999999992e-06 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929405+0000", 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041293140000000002 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929423+0000", 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8223000000000002e-05 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 46/32 ECSubWrite(tid=48, reqid=osd.14.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925520+0000", 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.83857246600000002, 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038787100000000001, 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925520+0000", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925520+0000", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925521+0000", 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2099999999999996e-07 2024-09-17T16:01:45.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925530+0000", 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925558+0000", 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7872e-05 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925565+0000", 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3180000000000001e-06 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925630+0000", 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5278999999999996e-05 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925640+0000", 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6420000000000002e-06 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929380+0000", 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037394989999999999 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929399+0000", 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9289000000000001e-05 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 46/32 ECSubWrite(tid=40, reqid=osd.0.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.82861710600000005, 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005398712, 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935476+0000", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6000000000000004e-07 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935486+0000", 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-09-17T16:01:45.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935508+0000", 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2333000000000001e-05 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935516+0000", 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8779999999999998e-06 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:45.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935579+0000", 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3529999999999997e-05 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935588+0000", 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9560000000000003e-06 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940834+0000", 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052456009999999999 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:45.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940874+0000", 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0224e-05 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:45.865 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.867 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:01:45.867 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:45.864+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:45.916 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.12 to 1 (10000) 2024-09-17T16:01:45.919 INFO:tasks.rados.rados.0.smithi181.stdout:101: done (3 left) 2024-09-17T16:01:45.919 INFO:tasks.rados.rados.0.smithi181.stdout:102: done (2 left) 2024-09-17T16:01:45.919 INFO:tasks.rados.rados.0.smithi181.stdout:103: done (1 left) 2024-09-17T16:01:45.919 INFO:tasks.rados.rados.0.smithi181.stdout:106: done (0 left) 2024-09-17T16:01:45.919 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 10 2024-09-17T16:01:45.920 INFO:tasks.rados.rados.0.smithi181.stdout:107: seq_num 67 ranges {1327104=753664,2080768=376832} 2024-09-17T16:01:45.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:01:45.930 INFO:tasks.rados.rados.0.smithi181.stdout:107: writing smithi18136759-24 from 1327104 to 2080768 tid 1 2024-09-17T16:01:45.932 INFO:tasks.rados.rados.0.smithi181.stdout:107: writing smithi18136759-24 from 2080768 to 2457600 tid 2 2024-09-17T16:01:45.932 INFO:tasks.rados.rados.0.smithi181.stdout:108: rollback oid 40 current snap is 10 2024-09-17T16:01:45.932 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 40 to 3 2024-09-17T16:01:45.933 INFO:tasks.rados.rados.0.smithi181.stdout:109: rollback oid 15 current snap is 10 2024-09-17T16:01:45.933 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 15 to 0 2024-09-17T16:01:45.933 INFO:tasks.rados.rados.0.smithi181.stdout:110: snap_remove snap 5 2024-09-17T16:01:45.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-17T16:01:45.975 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:45.973+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:45.975 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:45.975 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:45.976 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:45.976 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:45.983 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:01:46.068 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:46.066+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:46.086 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:46.086 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:46.086 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:46.086 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:46.086 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:46.095 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:01:46.169 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:46.168+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:46.199 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:46.199 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:8 3.0s0 3:03e99693:::smithi18136759-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.344022442, 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011147927, 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.200 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 8 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:46.201 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852744+0000", 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.159e-06 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852751+0000", 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9120000000000001e-06 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852762+0000", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1192000000000001e-05 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4889999999999998e-06 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854588+0000", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018183909999999999 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.203 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854806+0000", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217347 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855286+0000", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048023199999999998 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857505+0000", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022190869999999998 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863759+0000", 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062543 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863891+0000", 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013181799999999999 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:46.205 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.343982104, 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:46.206 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:01:46.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.209 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.210 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.343953591, 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.211 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:01:46.212 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.213 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:01:46.214 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:46.216 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.343814546000001, 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:01:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:01:46.221 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.343786872000001, 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.222 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:01:46.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:01:46.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs2 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.337979684, 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012875312, 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.227 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6400000000000001e-07 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858933+0000", 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014714799999999999 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858998+0000", 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5139000000000001e-05 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859014+0000", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5814000000000001e-05 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859094+0000", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9672000000000006e-05 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859104+0000", 2024-09-17T16:01:46.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0596e-05 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871628+0000", 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012524086 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871661+0000", 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2693e-05 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.231 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:39 3.9s0 3:905994b3:::smithi18136759-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.337807022, 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013438131000000001, 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 39 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.232 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858959+0000", 2024-09-17T16:01:46.233 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3399999999999998e-07 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858963+0000", 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.427e-06 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858970+0000", 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5270000000000001e-06 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.234 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858977+0000", 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9399999999999996e-06 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861610+0000", 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026333149999999998 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861663+0000", 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2731e-05 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.235 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868368+0000", 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067046069999999996 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868428+0000", 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0503e-05 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.236 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.872397+0000", 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039682470000000003 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=2, last_complete=34'3, committed=1, applied=1))", 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.333210628, 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014226245, 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.237 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:46.238 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863557+0000", 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1169999999999998e-06 2024-09-17T16:01:46.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863638+0000", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.03e-05 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863644+0000", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4150000000000004e-06 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877622+0000", 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013978305 2024-09-17T16:01:46.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877781+0000", 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015883299999999999 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.333206958, 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014337127, 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.242 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863559+0000", 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5800000000000001e-07 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863560+0000", 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863674+0000", 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011379000000000001 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863682+0000", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877875+0000", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014193012 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877896+0000", 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0993e-05 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.245 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s4 35/32 ECSubWrite(tid=119, reqid=client.4560.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.332068137, 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015598302, 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.246 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864698+0000", 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.44e-07 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.247 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864708+0000", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.73e-06 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864735+0000", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7804e-05 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864741+0000", 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7969999999999997e-06 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877824+0000", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01308231 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877833+0000", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4809999999999995e-06 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880272+0000", 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024384799999999998 2024-09-17T16:01:46.249 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880296+0000", 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4056000000000001e-05 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.279350294, 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.250 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:46.251 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:01:46.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:01:46.253 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:01:46.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.27932509, 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.255 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.256 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.257 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:46.258 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.259 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.275959095999999, 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.260 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:01:46.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:01:46.262 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.263 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.248984447, 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.264 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.265 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:01:46.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:01:46.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:01:46.268 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.248966576000001, 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.269 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:46.270 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.271 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:01:46.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:46.274 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.245672809, 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:46.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.276 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:01:46.277 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:185 3.9s0 3:99c5a4ac:::smithi18136759-42:head [copy-from ver 7] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.3238346009999997, 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019207894999999999, 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.278 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 185 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.279 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872932+0000", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.635e-06 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872938+0000", 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1680000000000003e-06 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872948+0000", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0477e-05 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872955+0000", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7750000000000004e-06 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873015+0000", 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:01:46.281 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873024+0000", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9439999999999994e-06 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874582+0000", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00155798 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874682+0000", 2024-09-17T16:01:46.282 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100215 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887716+0000", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013033741999999999 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889595+0000", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018794879999999999 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892089+0000", 2024-09-17T16:01:46.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024933450000000001 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892139+0000", 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0172e-05 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:186 3.9s0 3:99c5a4ac:::smithi18136759-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:46.284 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.3237979660000008, 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016243253999999999, 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 186 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:46.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872968+0000", 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2500000000000001e-07 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.286 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872969+0000", 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.525e-06 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872975+0000", 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.186e-06 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872978+0000", 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.202e-06 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873075+0000", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7590999999999994e-05 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873079+0000", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.54e-06 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874867+0000", 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017885100000000001 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for blocked object", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874930+0000", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2175999999999996e-05 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889013+0000", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014083304 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889112+0000", 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8972e-05 2024-09-17T16:01:46.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889211+0000", 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8923000000000001e-05 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 3.2455570539999998, 2024-09-17T16:01:46.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:46.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.2657036740000001, 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:46.294 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:46.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.297 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.298 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.298 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:46.298 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:46.298 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:46.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:01:46.300 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:46.297+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:46.399 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:46.397+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:46.414 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:46.415 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:46.415 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:46.415 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:46.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:01:46.500 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:46.498+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:46.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:46.527 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:46.527 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:46.527 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:01:46.527 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:01:46.527 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:46.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:01:46.601 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:46.599+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:46.639 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=166, reqid=client.4560.0:227, at_version=44'13, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.907558+0000", 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7293018930000001, 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002647234, 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907558+0000", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907558+0000", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907559+0000", 2024-09-17T16:01:46.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.71e-07 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907583+0000", 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4173999999999999e-05 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907629+0000", 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5708999999999999e-05 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907639+0000", 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0005e-05 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907702+0000", 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3589000000000004e-05 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.907714+0000", 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1355000000000001e-05 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.910168+0000", 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024537980000000001 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.910205+0000", 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7732999999999999e-05 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=167, reqid=client.4560.0:228, at_version=44'14, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:01:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7283788910000002, 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045964070000000003, 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908482+0000", 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908499+0000", 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6748999999999998e-05 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908533+0000", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3831999999999997e-05 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908539+0000", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5980000000000002e-06 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908619+0000", 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740999999999994e-05 2024-09-17T16:01:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908629+0000", 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913043+0000", 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044149380000000002 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913078+0000", 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4094999999999998e-05 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=168, reqid=client.4560.0:228, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.913782+0000", 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.723078251, 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019940790000000002, 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913782+0000", 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913782+0000", 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913783+0000", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1900000000000001e-07 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913793+0000", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0706e-05 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913832+0000", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.8368000000000003e-05 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913839+0000", 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0450000000000003e-06 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913894+0000", 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5714000000000002e-05 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913903+0000", 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.585e-06 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.915747+0000", 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0018435599999999999 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.915776+0000", 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9281999999999998e-05 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 44/32 ECSubWrite(tid=7, reqid=client.4560.0:238, at_version=44'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.922075+0000", 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7147848030000001, 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001991498, 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922075+0000", 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922075+0000", 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922076+0000", 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2500000000000001e-07 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922107+0000", 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0979000000000003e-05 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922212+0000", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010530399999999999 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922226+0000", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3763e-05 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922307+0000", 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1842999999999996e-05 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.922324+0000", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6249999999999999e-05 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.924032+0000", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001708141 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.924067+0000", 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4892999999999999e-05 2024-09-17T16:01:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 44/32 ECSubWrite(tid=8, reqid=client.4560.0:239, at_version=44'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.923426+0000", 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7134344829999999, 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001889513, 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923426+0000", 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923426+0000", 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923426+0000", 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3900000000000003e-07 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923448+0000", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1716e-05 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923664+0000", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021545000000000001 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923677+0000", 2024-09-17T16:01:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2886e-05 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923738+0000", 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1661999999999993e-05 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.923748+0000", 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7569999999999999e-06 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.925261+0000", 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001513431 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.925315+0000", 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3671999999999997e-05 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7086627170000002, 2024-09-17T16:01:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003430879, 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928198+0000", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2200000000000002e-07 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928663+0000", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046509200000000001 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928719+0000", 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5677000000000002e-05 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928728+0000", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6729999999999998e-06 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928810+0000", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2386999999999996e-05 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928821+0000", 2024-09-17T16:01:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0797e-05 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931371+0000", 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002550549 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931628+0000", 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000256782 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=35, reqid=client.4560.0:242, at_version=44'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.930125+0000", 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7067349090000001, 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0030249040000000001, 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930125+0000", 2024-09-17T16:01:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930125+0000", 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930126+0000", 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4099999999999997e-07 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930355+0000", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00022922200000000001 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930405+0000", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9892999999999998e-05 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930417+0000", 2024-09-17T16:01:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2118e-05 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930482+0000", 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4855000000000002e-05 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.930492+0000", 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4439999999999997e-06 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.933097+0000", 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002604968 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.933150+0000", 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3463e-05 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=36, reqid=client.4560.0:243, at_version=44'13, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.931328+0000", 2024-09-17T16:01:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 3.7055315819999999, 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022865849999999998, 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931328+0000", 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931328+0000", 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931329+0000", 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5199999999999996e-07 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931349+0000", 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9306000000000001e-05 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931382+0000", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3238000000000003e-05 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931398+0000", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6021000000000001e-05 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931735+0000", 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033751400000000002 2024-09-17T16:01:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931745+0000", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4809999999999995e-06 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.933578+0000", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00183304 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.933615+0000", 2024-09-17T16:01:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7233000000000001e-05 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7101796609999997, 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031449350000000002, 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926681+0000", 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.22e-07 2024-09-17T16:01:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926710+0000", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9631000000000001e-05 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926780+0000", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9764000000000005e-05 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926797+0000", 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7201999999999999e-05 2024-09-17T16:01:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926888+0000", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0939000000000005e-05 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926902+0000", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3997e-05 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929791+0000", 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028892620000000001 2024-09-17T16:01:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929825+0000", 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3918000000000003e-05 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7091157099999998, 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046997530000000001, 2024-09-17T16:01:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927762+0000", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7229000000000001e-05 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927945+0000", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018362200000000001 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928161+0000", 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021584100000000001 2024-09-17T16:01:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928170+0000", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6249999999999996e-06 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928242+0000", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2434999999999993e-05 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928253+0000", 2024-09-17T16:01:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932409+0000", 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041564169999999999 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932444+0000", 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7089106840000001, 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048629329999999998, 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927950+0000", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3399999999999999e-07 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928041+0000", 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0953999999999997e-05 2024-09-17T16:01:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928205+0000", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000164537 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928213+0000", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.272e-06 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929483+0000", 2024-09-17T16:01:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001270106 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929493+0000", 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0207000000000001e-05 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932778+0000", 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032847419999999998 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932812+0000", 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4581e-05 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=12, reqid=client.4560.0:273, at_version=45'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7087791650000002, 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043524990000000001, 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928082+0000", 2024-09-17T16:01:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2799999999999996e-07 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928098+0000", 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6070999999999999e-05 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928229+0000", 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013130999999999999 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928234+0000", 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928583+0000", 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034869300000000002 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928595+0000", 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1433e-05 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932398+0000", 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038034890000000002 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932433+0000", 2024-09-17T16:01:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5386999999999998e-05 2024-09-17T16:01:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7081187780000002, 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003718285, 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:01:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928742+0000", 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928761+0000", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8666000000000001e-05 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928789+0000", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8535999999999999e-05 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928798+0000", 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4479999999999994e-06 2024-09-17T16:01:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928947+0000", 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000149068 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928956+0000", 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0799999999999995e-06 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932437+0000", 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034817609999999999 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932460+0000", 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2078000000000001e-05 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.930166+0000", 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 2.7066943559999999, 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002670801, 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930166+0000", 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930166+0000", 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930167+0000", 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3500000000000005e-07 2024-09-17T16:01:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930186+0000", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9766e-05 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930233+0000", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6258999999999999e-05 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930242+0000", 2024-09-17T16:01:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6029999999999994e-06 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930296+0000", 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4069e-05 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930302+0000", 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8919999999999997e-06 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932817+0000", 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025147989999999999 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932836+0000", 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9477999999999999e-05 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/32 ECSubWrite(tid=182, reqid=osd.11.0:182, at_version=46'20, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.917739+0000", 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.7191214559999999, 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021969469999999999, 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917739+0000", 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917739+0000", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917739+0000", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9900000000000001e-07 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917768+0000", 2024-09-17T16:01:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9493999999999999e-05 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917825+0000", 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6094999999999997e-05 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917835+0000", 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0837e-05 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917946+0000", 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000110463 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.917957+0000", 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0713999999999999e-05 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919899+0000", 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001942378 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919936+0000", 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6566999999999997e-05 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.718317428, 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039399220000000002, 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.29e-07 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5406999999999999e-05 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102031 2024-09-17T16:01:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918679+0000", 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918809+0000", 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012967600000000001 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918820+0000", 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1156e-05 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922452+0000", 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036319080000000001 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922483+0000", 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0338999999999999e-05 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.71789133, 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035116600000000002, 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918994+0000", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4694000000000001e-05 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919035+0000", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1223000000000002e-05 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919044+0000", 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0599999999999997e-06 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919108+0000", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3326999999999999e-05 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919118+0000", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0037999999999999e-05 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922439+0000", 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033215340000000001 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922480+0000", 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1010000000000002e-05 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.7172624810000001, 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043836329999999996, 2024-09-17T16:01:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919621+0000", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2204e-05 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919652+0000", 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1846e-05 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919661+0000", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3075000000000002e-05 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919733+0000", 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3540000000000006e-06 2024-09-17T16:01:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923937+0000", 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042032980000000003 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923981+0000", 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4685000000000003e-05 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 46/32 ECSubWrite(tid=16, reqid=osd.1.0:15, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.921427+0000", 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.7154333720000001, 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0024499560000000001, 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921427+0000", 2024-09-17T16:01:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921427+0000", 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921428+0000", 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921440+0000", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2695e-05 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921478+0000", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7540000000000003e-05 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921485+0000", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2250000000000003e-06 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921565+0000", 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9534000000000001e-05 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921576+0000", 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1404e-05 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923837+0000", 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002260986 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923877+0000", 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9681999999999997e-05 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.as3 47/32 ECSubWrite(tid=61, reqid=client.4560.0:284, at_version=47'20, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:45.939189+0000", 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.69767127900000003, 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023078439999999999, 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:01:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939189+0000", 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939189+0000", 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939190+0000", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.02e-06 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939222+0000", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2335999999999997e-05 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939261+0000", 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9314999999999999e-05 2024-09-17T16:01:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939271+0000", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6029999999999994e-06 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939335+0000", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4128999999999995e-05 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.939346+0000", 2024-09-17T16:01:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1171e-05 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.941453+0000", 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021071430000000001 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.941497+0000", 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3127000000000002e-05 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:01:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:01:46.734 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:46.735 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:46.735 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:46.736 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:46.736 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:46.736 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:01:46.737 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:46.737 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:46.737 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:46.738 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:46.738 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:46.732+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:46.741 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:46.741 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:46.763 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:46.763 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:46.784 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:46.785 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:46.806 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:46.806 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:46.827 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:46.827 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:46.835 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:46.833+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:46.840 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:46.841 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:46.849 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:46.849 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:46.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-17T16:01:46.870 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:46.870 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:46.892 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:46.893 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:46.912 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:46.912 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:46.924 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:01:46.925 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.13 to 1 (10000) 2024-09-17T16:01:46.931 INFO:tasks.rados.rados.0.smithi181.stdout:110: done (3 left) 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:111: rmattr oid 39 current snap is 10 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:111: done (3 left) 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:112: copy_from oid 5 from oid 1 current snap is 10 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:107: finishing write tid 1 to smithi18136759-24 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:108: finishing rollback tid 0 to smithi18136759-40 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:109: finishing rollback tid 0 to smithi18136759-15 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:107: finishing write tid 2 to smithi18136759-24 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:107: finishing write tid 3 to smithi18136759-24 2024-09-17T16:01:46.932 INFO:tasks.rados.rados.0.smithi181.stdout:108: finishing rollback tid 1 to smithi18136759-40 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:109: finishing rollback tid 1 to smithi18136759-15 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:109: finishing rollback tid 2 to smithi18136759-15 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 16 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: finishing write tid 4 to smithi18136759-24 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: oid 24 updating version 0 to 22 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: oid 24 updating version 22 to 23 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: oid 24 version 23 is already newer than 21 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 23 (ObjNum 67 snap 10 seq_num 67) dirty exists 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: left oid 24 (ObjNum 67 snap 10 seq_num 67) 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:108: finishing rollback tid 2 to smithi18136759-40 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 20 (ObjNum 52 snap 2 seq_num 52) dirty exists 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:107: done (3 left) 2024-09-17T16:01:46.933 INFO:tasks.rados.rados.0.smithi181.stdout:108: done (2 left) 2024-09-17T16:01:46.934 INFO:tasks.rados.rados.0.smithi181.stdout:109: done (1 left) 2024-09-17T16:01:46.934 INFO:tasks.rados.rados.0.smithi181.stdout:113: read oid 47 snap -1 2024-09-17T16:01:46.934 INFO:tasks.rados.rados.0.smithi181.stdout:113: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-17T16:01:46.934 INFO:tasks.rados.rados.0.smithi181.stdout:114: copy_from oid 10 from oid 38 current snap is 10 2024-09-17T16:01:46.934 INFO:tasks.rados.rados.0.smithi181.stdout:115: snap_create 2024-09-17T16:01:46.936 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:46.933+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:46.939 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing copy_from to smithi18136759-5 2024-09-17T16:01:46.939 INFO:tasks.rados.rados.0.smithi181.stdout:112: got expected ENOENT (src dne) 2024-09-17T16:01:46.939 INFO:tasks.rados.rados.0.smithi181.stdout:112: finishing copy_from racing read to smithi18136759-5 2024-09-17T16:01:46.940 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:46.940 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:46.945 INFO:tasks.rados.rados.0.smithi181.stdout:113: expect (ObjNum 46 snap 0 seq_num 46) 2024-09-17T16:01:46.945 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:46.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-17T16:01:46.957 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 515692 2024-09-17T16:01:46.960 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:46.967 INFO:tasks.rados.rados.0.smithi181.stdout:114: finishing copy_from racing read to smithi18136759-10 2024-09-17T16:01:46.967 INFO:tasks.rados.rados.0.smithi181.stdout:114: finishing copy_from to smithi18136759-10 2024-09-17T16:01:46.967 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 8 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:01:47.002 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.002 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-17T16:01:47.051 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:47.049+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:47.137 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:47.135+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:47.149 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:47.147+0000 7f961d23e640 0 -- 172.21.15.6:0/4284268506 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7f961821f430 msgr2=0x7f9618221820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:47.153 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.154 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:01:47.170 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:47.171 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:47.239 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:47.236+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:47.305 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.308 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:01:47.331 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.14 to 1 (10000) 2024-09-17T16:01:47.334 INFO:tasks.rados.rados.0.smithi181.stdout:112: done (3 left) 2024-09-17T16:01:47.334 INFO:tasks.rados.rados.0.smithi181.stdout:113: done (2 left) 2024-09-17T16:01:47.335 INFO:tasks.rados.rados.0.smithi181.stdout:114: done (1 left) 2024-09-17T16:01:47.335 INFO:tasks.rados.rados.0.smithi181.stdout:115: done (0 left) 2024-09-17T16:01:47.335 INFO:tasks.rados.rados.0.smithi181.stdout:116: rollback oid 44 current snap is 11 2024-09-17T16:01:47.335 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 44 to 10 2024-09-17T16:01:47.335 INFO:tasks.rados.rados.0.smithi181.stdout:117: snap_create 2024-09-17T16:01:47.339 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:47.337+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:47.339 INFO:tasks.rados.rados.0.smithi181.stdout:116: finishing rollback tid 0 to smithi18136759-44 2024-09-17T16:01:47.342 INFO:tasks.rados.rados.0.smithi181.stdout:116: finishing rollback tid 1 to smithi18136759-44 2024-09-17T16:01:47.342 INFO:tasks.rados.rados.0.smithi181.stdout:116: finishing rollback tid 2 to smithi18136759-44 2024-09-17T16:01:47.343 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 28 (ObjNum 59 snap 7 seq_num 59) dirty exists 2024-09-17T16:01:47.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-17T16:01:47.420 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.420 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:01:47.440 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:47.438+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:47.541 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:47.539+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:47.573 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.573 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:01:47.643 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:47.641+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:47.724 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.724 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:01:47.743 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:47.740+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:47.834 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.835 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:01:47.845 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:47.843+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:47.946 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:47.944+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:47.946 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:47.947 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:47.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:01:47.964 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:47.965 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:48.047 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:48.045+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:48.067 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.067 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:01:48.160 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:48.158+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:48.224 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.224 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:01:48.274 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:48.272+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:48.333 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.15 to 1 (10000) 2024-09-17T16:01:48.336 INFO:tasks.rados.rados.0.smithi181.stdout:116: done (1 left) 2024-09-17T16:01:48.336 INFO:tasks.rados.rados.0.smithi181.stdout:117: done (0 left) 2024-09-17T16:01:48.336 INFO:tasks.rados.rados.0.smithi181.stdout:118: read oid 26 snap -1 2024-09-17T16:01:48.336 INFO:tasks.rados.rados.0.smithi181.stdout:118: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-17T16:01:48.336 INFO:tasks.rados.rados.0.smithi181.stdout:119: rmattr oid 23 current snap is 12 2024-09-17T16:01:48.337 INFO:tasks.rados.rados.0.smithi181.stdout:119: done (1 left) 2024-09-17T16:01:48.337 INFO:tasks.rados.rados.0.smithi181.stdout:120: setattr oid 43 current snap is 12 2024-09-17T16:01:48.337 INFO:tasks.rados.rados.0.smithi181.stdout:121: read oid 10 snap -1 2024-09-17T16:01:48.337 INFO:tasks.rados.rados.0.smithi181.stdout:121: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T16:01:48.337 INFO:tasks.rados.rados.0.smithi181.stdout:122: delete oid 48 current snap is 12 2024-09-17T16:01:48.344 INFO:tasks.rados.rados.0.smithi181.stdout:118: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-17T16:01:48.344 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:48.350 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:48.348+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:48.350 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 344620 2024-09-17T16:01:48.351 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:48.353 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:01:48.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:48.355 INFO:tasks.rados.rados.0.smithi181.stdout:121: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T16:01:48.357 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 349295 2024-09-17T16:01:48.359 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 280639 2024-09-17T16:01:48.360 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 43 v 16 (ObjNum 58 snap 6 seq_num 58) dirty exists 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:118: done (3 left) 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:120: done (2 left) 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:121: done (1 left) 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:122: done (0 left) 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 12 2024-09-17T16:01:48.365 INFO:tasks.rados.rados.0.smithi181.stdout:123: seq_num 69 ranges {2457600=786432,3244032=376832} 2024-09-17T16:01:48.382 INFO:tasks.rados.rados.0.smithi181.stdout:123: writing smithi18136759-24 from 2457600 to 3244032 tid 1 2024-09-17T16:01:48.384 INFO:tasks.rados.rados.0.smithi181.stdout:123: writing smithi18136759-24 from 3244032 to 3620864 tid 2 2024-09-17T16:01:48.384 INFO:tasks.rados.rados.0.smithi181.stdout:124: copy_from oid 37 from oid 22 current snap is 12 2024-09-17T16:01:48.384 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 12 2024-09-17T16:01:48.384 INFO:tasks.rados.rados.0.smithi181.stdout:125: seq_num 70 ranges {835584=802816,1638400=376832} 2024-09-17T16:01:48.391 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.391 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.393 INFO:tasks.rados.rados.0.smithi181.stdout:125: writing smithi18136759-50 from 835584 to 1638400 tid 1 2024-09-17T16:01:48.395 INFO:tasks.rados.rados.0.smithi181.stdout:125: writing smithi18136759-50 from 1638400 to 2015232 tid 2 2024-09-17T16:01:48.395 INFO:tasks.rados.rados.0.smithi181.stdout:126: snap_create 2024-09-17T16:01:48.395 INFO:tasks.rados.rados.0.smithi181.stdout:123: finishing write tid 1 to smithi18136759-24 2024-09-17T16:01:48.395 INFO:tasks.rados.rados.0.smithi181.stdout:123: finishing write tid 2 to smithi18136759-24 2024-09-17T16:01:48.398 INFO:tasks.rados.rados.0.smithi181.stdout:125: finishing write tid 1 to smithi18136759-50 2024-09-17T16:01:48.400 INFO:tasks.rados.rados.0.smithi181.stdout:125: finishing write tid 2 to smithi18136759-50 2024-09-17T16:01:48.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: finishing write tid 3 to smithi18136759-50 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: finishing write tid 4 to smithi18136759-50 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: oid 50 updating version 0 to 24 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: oid 50 updating version 24 to 25 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: oid 50 updating version 25 to 26 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 26 (ObjNum 70 snap 12 seq_num 70) dirty exists 2024-09-17T16:01:48.402 INFO:tasks.rados.rados.0.smithi181.stdout:125: left oid 50 (ObjNum 70 snap 12 seq_num 70) 2024-09-17T16:01:48.451 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:48.449+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:48.551 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.551 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.552 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:48.550+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:48.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:01:48.582 INFO:tasks.rados.rados.0.smithi181.stdout:123: finishing write tid 3 to smithi18136759-24 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:123: finishing write tid 4 to smithi18136759-24 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:123: oid 24 updating version 0 to 29 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:123: oid 24 updating version 29 to 31 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:123: oid 24 version 31 is already newer than 30 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 31 (ObjNum 69 snap 12 seq_num 69) dirty exists 2024-09-17T16:01:48.583 INFO:tasks.rados.rados.0.smithi181.stdout:123: left oid 24 (ObjNum 69 snap 12 seq_num 69) 2024-09-17T16:01:48.597 INFO:tasks.rados.rados.0.smithi181.stdout:124: finishing copy_from racing read to smithi18136759-37 2024-09-17T16:01:48.601 INFO:tasks.rados.rados.0.smithi181.stdout:124: finishing copy_from to smithi18136759-37 2024-09-17T16:01:48.601 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 4 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-17T16:01:48.653 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:48.650+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:48.666 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:48.678 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-17T16:01:48.332525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37749299,"num_objects":79,"num_object_clones":27,"num_object_copies":533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":1,"num_read":251,"num_read_kb":9129,"num_write":348,"num_write_kb":38290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":325,"ondisk_log_size":325,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":508284,"kb_used_data":67444,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509441156,"statfs":{"total":1546188226560,"available":1545667743744,"internally_reserved":0,"allocated":69062656,"data_stored":84202997,"data_compressed":1122424,"data_compressed_allocated":1576960,"data_compressed_original":4030464,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":1370,"num_write":8,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.421375"},"pg_stats":[{"pgid":"3.f","version":"44'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921616+0000","last_change":"2024-09-17T16:01:37.883270+0000","last_active":"2024-09-17T16:01:43.921616+0000","last_peered":"2024-09-17T16:01:43.921616+0000","last_clean":"2024-09-17T16:01:43.921616+0000","last_became_active":"2024-09-17T16:01:31.980473+0000","last_became_peered":"2024-09-17T16:01:31.980473+0000","last_unstale":"2024-09-17T16:01:43.921616+0000","last_undegraded":"2024-09-17T16:01:43.921616+0000","last_fullsized":"2024-09-17T16:01:43.921616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:53.770253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052621489999999998,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":367,"num_write":8,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.e","version":"45'16","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931984+0000","last_change":"2024-09-17T16:01:37.870258+0000","last_active":"2024-09-17T16:01:43.931984+0000","last_peered":"2024-09-17T16:01:43.931984+0000","last_clean":"2024-09-17T16:01:43.931984+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:01:43.931984+0000","last_undegraded":"2024-09-17T16:01:43.931984+0000","last_fullsized":"2024-09-17T16:01:43.931984+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.879288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045074600000000002,"stat_sum":{"num_bytes":3620864,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":2454,"num_write":33,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.d","version":"47'23","reported_seq":100,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944274+0000","last_change":"2024-09-17T16:01:45.879561+0000","last_active":"2024-09-17T16:01:45.944274+0000","last_peered":"2024-09-17T16:01:45.944274+0000","last_clean":"2024-09-17T16:01:45.944274+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:01:45.944274+0000","last_undegraded":"2024-09-17T16:01:45.944274+0000","last_fullsized":"2024-09-17T16:01:45.944274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.856272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95719047499999999,"stat_sum":{"num_bytes":3457024,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1376,"num_write":19,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.c","version":"44'12","reported_seq":67,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921602+0000","last_change":"2024-09-17T16:01:37.878565+0000","last_active":"2024-09-17T16:01:43.921602+0000","last_peered":"2024-09-17T16:01:43.921602+0000","last_clean":"2024-09-17T16:01:43.921602+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:01:43.921602+0000","last_undegraded":"2024-09-17T16:01:43.921602+0000","last_fullsized":"2024-09-17T16:01:43.921602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.083328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000730739,"stat_sum":{"num_bytes":1933312,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":9,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.b","version":"45'12","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921162+0000","last_change":"2024-09-17T16:01:37.890455+0000","last_active":"2024-09-17T16:01:43.921162+0000","last_peered":"2024-09-17T16:01:43.921162+0000","last_clean":"2024-09-17T16:01:43.921162+0000","last_became_active":"2024-09-17T16:01:32.027225+0000","last_became_peered":"2024-09-17T16:01:32.027225+0000","last_unstale":"2024-09-17T16:01:43.921162+0000","last_undegraded":"2024-09-17T16:01:43.921162+0000","last_fullsized":"2024-09-17T16:01:43.921162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:35.171662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080637500000000002,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":816,"num_write":9,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,7],"acting":[13,1,11,15,6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.a","version":"47'20","reported_seq":85,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944277+0000","last_change":"2024-09-17T16:01:44.938624+0000","last_active":"2024-09-17T16:01:45.944277+0000","last_peered":"2024-09-17T16:01:45.944277+0000","last_clean":"2024-09-17T16:01:45.944277+0000","last_became_active":"2024-09-17T16:01:32.579608+0000","last_became_peered":"2024-09-17T16:01:32.579608+0000","last_unstale":"2024-09-17T16:01:45.944277+0000","last_undegraded":"2024-09-17T16:01:45.944277+0000","last_fullsized":"2024-09-17T16:01:45.944277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:57.246615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016324966999999999,"stat_sum":{"num_bytes":3293184,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,6,8],"acting":[12,4,7,3,14,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.9","version":"44'15","reported_seq":74,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947381+0000","last_change":"2024-09-17T16:01:37.875257+0000","last_active":"2024-09-17T16:01:43.947381+0000","last_peered":"2024-09-17T16:01:43.947381+0000","last_clean":"2024-09-17T16:01:43.947381+0000","last_became_active":"2024-09-17T16:01:32.027179+0000","last_became_peered":"2024-09-17T16:01:32.027179+0000","last_unstale":"2024-09-17T16:01:43.947381+0000","last_undegraded":"2024-09-17T16:01:43.947381+0000","last_fullsized":"2024-09-17T16:01:43.947381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.899899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085827199999999998,"stat_sum":{"num_bytes":4128768,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":13,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.8","version":"44'25","reported_seq":81,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921816+0000","last_change":"2024-09-17T16:01:37.884652+0000","last_active":"2024-09-17T16:01:43.921816+0000","last_peered":"2024-09-17T16:01:43.921816+0000","last_clean":"2024-09-17T16:01:43.921816+0000","last_became_active":"2024-09-17T16:01:31.848773+0000","last_became_peered":"2024-09-17T16:01:31.848773+0000","last_unstale":"2024-09-17T16:01:43.921816+0000","last_undegraded":"2024-09-17T16:01:43.921816+0000","last_fullsized":"2024-09-17T16:01:43.921816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:55.083415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096304449999999996,"stat_sum":{"num_bytes":3784704,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":15,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":189,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921662+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:43.921662+0000","last_peered":"2024-09-17T16:01:43.921662+0000","last_clean":"2024-09-17T16:01:43.921662+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:43.921662+0000","last_undegraded":"2024-09-17T16:01:43.921662+0000","last_fullsized":"2024-09-17T16:01:43.921662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":55,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921551+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:43.921551+0000","last_peered":"2024-09-17T16:01:43.921551+0000","last_clean":"2024-09-17T16:01:43.921551+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:43.921551+0000","last_undegraded":"2024-09-17T16:01:43.921551+0000","last_fullsized":"2024-09-17T16:01:43.921551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'18","reported_seq":86,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.940205+0000","last_change":"2024-09-17T16:01:37.875712+0000","last_active":"2024-09-17T16:01:43.940205+0000","last_peered":"2024-09-17T16:01:43.940205+0000","last_clean":"2024-09-17T16:01:43.940205+0000","last_became_active":"2024-09-17T16:01:32.579280+0000","last_became_peered":"2024-09-17T16:01:32.579280+0000","last_unstale":"2024-09-17T16:01:43.940205+0000","last_undegraded":"2024-09-17T16:01:43.940205+0000","last_fullsized":"2024-09-17T16:01:43.940205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:40.033715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078007100000000002,"stat_sum":{"num_bytes":3112960,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":22,"num_read_kb":2062,"num_write":14,"num_write_kb":2960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929150+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:43.929150+0000","last_peered":"2024-09-17T16:01:43.929150+0000","last_clean":"2024-09-17T16:01:43.929150+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:43.929150+0000","last_undegraded":"2024-09-17T16:01:43.929150+0000","last_fullsized":"2024-09-17T16:01:43.929150+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"35'6","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.920904+0000","last_change":"2024-09-17T16:01:37.890929+0000","last_active":"2024-09-17T16:01:43.920904+0000","last_peered":"2024-09-17T16:01:43.920904+0000","last_clean":"2024-09-17T16:01:43.920904+0000","last_became_active":"2024-09-17T16:01:32.579478+0000","last_became_peered":"2024-09-17T16:01:32.579478+0000","last_unstale":"2024-09-17T16:01:43.920904+0000","last_undegraded":"2024-09-17T16:01:43.920904+0000","last_fullsized":"2024-09-17T16:01:43.920904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.792345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091054000000000005,"stat_sum":{"num_bytes":1245184,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.930873+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:43.930873+0000","last_peered":"2024-09-17T16:01:43.930873+0000","last_clean":"2024-09-17T16:01:43.930873+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:43.930873+0000","last_undegraded":"2024-09-17T16:01:43.930873+0000","last_fullsized":"2024-09-17T16:01:43.930873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"44'15","reported_seq":75,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947349+0000","last_change":"2024-09-17T16:01:37.874906+0000","last_active":"2024-09-17T16:01:43.947349+0000","last_peered":"2024-09-17T16:01:43.947349+0000","last_clean":"2024-09-17T16:01:43.947349+0000","last_became_active":"2024-09-17T16:01:32.578748+0000","last_became_peered":"2024-09-17T16:01:32.578748+0000","last_unstale":"2024-09-17T16:01:43.947349+0000","last_undegraded":"2024-09-17T16:01:43.947349+0000","last_fullsized":"2024-09-17T16:01:43.947349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:31.899666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051024999999999996,"stat_sum":{"num_bytes":2113536,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":135,"num_write":31,"num_write_kb":2064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":56,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929283+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:43.929283+0000","last_peered":"2024-09-17T16:01:43.929283+0000","last_clean":"2024-09-17T16:01:43.929283+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:43.929283+0000","last_undegraded":"2024-09-17T16:01:43.929283+0000","last_fullsized":"2024-09-17T16:01:43.929283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"39'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931043+0000","last_change":"2024-09-17T16:01:37.886987+0000","last_active":"2024-09-17T16:01:43.931043+0000","last_peered":"2024-09-17T16:01:43.931043+0000","last_clean":"2024-09-17T16:01:43.931043+0000","last_became_active":"2024-09-17T16:01:32.027376+0000","last_became_peered":"2024-09-17T16:01:32.027376+0000","last_unstale":"2024-09-17T16:01:43.931043+0000","last_undegraded":"2024-09-17T16:01:43.931043+0000","last_fullsized":"2024-09-17T16:01:43.931043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.273009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069738480000000004,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.5","version":"35'4","reported_seq":61,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929570+0000","last_change":"2024-09-17T16:01:37.885001+0000","last_active":"2024-09-17T16:01:43.929570+0000","last_peered":"2024-09-17T16:01:43.929570+0000","last_clean":"2024-09-17T16:01:43.929570+0000","last_became_active":"2024-09-17T16:01:31.981156+0000","last_became_peered":"2024-09-17T16:01:31.981156+0000","last_unstale":"2024-09-17T16:01:43.929570+0000","last_undegraded":"2024-09-17T16:01:43.929570+0000","last_fullsized":"2024-09-17T16:01:43.929570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.852744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066925399999999999,"stat_sum":{"num_bytes":1130496,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-17T16:01:42.919942+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:42.919942+0000","last_peered":"2024-09-17T16:01:42.919942+0000","last_clean":"2024-09-17T16:01:42.919942+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:42.919942+0000","last_undegraded":"2024-09-17T16:01:42.919942+0000","last_fullsized":"2024-09-17T16:01:42.919942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"44'8","reported_seq":61,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-17T16:01:42.932584+0000","last_change":"2024-09-17T16:01:37.877638+0000","last_active":"2024-09-17T16:01:42.932584+0000","last_peered":"2024-09-17T16:01:42.932584+0000","last_clean":"2024-09-17T16:01:42.932584+0000","last_became_active":"2024-09-17T16:01:32.579546+0000","last_became_peered":"2024-09-17T16:01:32.579546+0000","last_unstale":"2024-09-17T16:01:42.932584+0000","last_undegraded":"2024-09-17T16:01:42.932584+0000","last_fullsized":"2024-09-17T16:01:42.932584+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.479900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045131899999999999,"stat_sum":{"num_bytes":1064960,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":6,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.906323+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:43.906323+0000","last_peered":"2024-09-17T16:01:43.906323+0000","last_clean":"2024-09-17T16:01:43.906323+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:43.906323+0000","last_undegraded":"2024-09-17T16:01:43.906323+0000","last_fullsized":"2024-09-17T16:01:43.906323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"42'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921053+0000","last_change":"2024-09-17T16:01:37.890423+0000","last_active":"2024-09-17T16:01:43.921053+0000","last_peered":"2024-09-17T16:01:43.921053+0000","last_clean":"2024-09-17T16:01:43.921053+0000","last_became_active":"2024-09-17T16:01:31.981906+0000","last_became_peered":"2024-09-17T16:01:31.981906+0000","last_unstale":"2024-09-17T16:01:43.921053+0000","last_undegraded":"2024-09-17T16:01:43.921053+0000","last_fullsized":"2024-09-17T16:01:43.921053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.018677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073874200000000002,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.928508+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:43.928508+0000","last_peered":"2024-09-17T16:01:43.928508+0000","last_clean":"2024-09-17T16:01:43.928508+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:43.928508+0000","last_undegraded":"2024-09-17T16:01:43.928508+0000","last_fullsized":"2024-09-17T16:01:43.928508+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"35'6","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921133+0000","last_change":"2024-09-17T16:01:37.890930+0000","last_active":"2024-09-17T16:01:43.921133+0000","last_peered":"2024-09-17T16:01:43.921133+0000","last_clean":"2024-09-17T16:01:43.921133+0000","last_became_active":"2024-09-17T16:01:31.981275+0000","last_became_peered":"2024-09-17T16:01:31.981275+0000","last_unstale":"2024-09-17T16:01:43.921133+0000","last_undegraded":"2024-09-17T16:01:43.921133+0000","last_fullsized":"2024-09-17T16:01:43.921133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:51.368632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089277400000000004,"stat_sum":{"num_bytes":1114112,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":1815,"num_write":6,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929359+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:43.929359+0000","last_peered":"2024-09-17T16:01:43.929359+0000","last_clean":"2024-09-17T16:01:43.929359+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:43.929359+0000","last_undegraded":"2024-09-17T16:01:43.929359+0000","last_fullsized":"2024-09-17T16:01:43.929359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37158912,"num_objects":75,"num_object_clones":27,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":1,"num_read":165,"num_read_kb":9056,"num_write":201,"num_write_kb":35616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61554688,"data_stored":79028224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247818,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30240,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341600,"statfs":{"total":96636764160,"available":96605798400,"internally_reserved":0,"allocated":3284992,"data_stored":4304096,"data_compressed":75108,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30652,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341188,"statfs":{"total":96636764160,"available":96605376512,"internally_reserved":0,"allocated":3706880,"data_stored":4689046,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":4320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":4423680,"data_stored":5528726,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35104,"kb_used_data":3908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336736,"statfs":{"total":96636764160,"available":96600817664,"internally_reserved":0,"allocated":4001792,"data_stored":4940941,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":5705728,"data_stored":6978710,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":5533696,"data_stored":6651030,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":4780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":4894720,"data_stored":6549174,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30124,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341716,"statfs":{"total":96636764160,"available":96605917184,"internally_reserved":0,"allocated":3166208,"data_stored":3673238,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":6164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":6311936,"data_stored":7478422,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30084,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341756,"statfs":{"total":96636764160,"available":96605958144,"internally_reserved":0,"allocated":3125248,"data_stored":3433613,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":4536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":4644864,"data_stored":6184086,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30728,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341112,"statfs":{"total":96636764160,"available":96605298688,"internally_reserved":0,"allocated":3784704,"data_stored":4463766,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30760,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341080,"statfs":{"total":96636764160,"available":96605265920,"internally_reserved":0,"allocated":3817472,"data_stored":4193449,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30020,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341820,"statfs":{"total":96636764160,"available":96606023680,"internally_reserved":0,"allocated":3063808,"data_stored":3232928,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29776,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342064,"statfs":{"total":96636764160,"available":96606273536,"internally_reserved":0,"allocated":2805760,"data_stored":2903190,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33660,"kb_used_data":6632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338180,"statfs":{"total":96636764160,"available":96602296320,"internally_reserved":0,"allocated":6791168,"data_stored":8998582,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5226496,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3534848,"data_stored":4657152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2686976,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5832704,"data_stored":7184384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":5890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3305472,"data_stored":4169728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3334144,"data_stored":3899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2592768,"data_stored":3055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2326528,"data_stored":2609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6254592,"data_stored":8175616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2781184,"data_stored":3989504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5054464,"data_stored":6356992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3944448,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3227648,"data_stored":4395008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:48.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:48.709 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.709 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:01:48.754 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:48.751+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:48.855 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:48.853+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:48.867 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.868 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:01:48.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:01:48.955 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:48.953+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:48.984 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:48.984 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:48.993 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:48.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:01:49.006 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-09-17T16:01:48.942292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37749299,"num_objects":80,"num_object_clones":28,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":1,"num_read":255,"num_read_kb":10042,"num_write":350,"num_write_kb":38290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":333,"ondisk_log_size":333,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":508356,"kb_used_data":67516,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509441084,"statfs":{"total":1546188226560,"available":1545667670016,"internally_reserved":0,"allocated":69136384,"data_stored":84858627,"data_compressed":1140662,"data_compressed_allocated":1601536,"data_compressed_original":4079616,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2129920,"num_objects":4,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":2283,"num_write":10,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000814"},"pg_stats":[{"pgid":"3.f","version":"44'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921616+0000","last_change":"2024-09-17T16:01:37.883270+0000","last_active":"2024-09-17T16:01:43.921616+0000","last_peered":"2024-09-17T16:01:43.921616+0000","last_clean":"2024-09-17T16:01:43.921616+0000","last_became_active":"2024-09-17T16:01:31.980473+0000","last_became_peered":"2024-09-17T16:01:31.980473+0000","last_unstale":"2024-09-17T16:01:43.921616+0000","last_undegraded":"2024-09-17T16:01:43.921616+0000","last_fullsized":"2024-09-17T16:01:43.921616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:53.770253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052621489999999998,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":367,"num_write":8,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.e","version":"45'16","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931984+0000","last_change":"2024-09-17T16:01:37.870258+0000","last_active":"2024-09-17T16:01:43.931984+0000","last_peered":"2024-09-17T16:01:43.931984+0000","last_clean":"2024-09-17T16:01:43.931984+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:01:43.931984+0000","last_undegraded":"2024-09-17T16:01:43.931984+0000","last_fullsized":"2024-09-17T16:01:43.931984+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.879288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045074600000000002,"stat_sum":{"num_bytes":3620864,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":2454,"num_write":33,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.d","version":"47'23","reported_seq":100,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944274+0000","last_change":"2024-09-17T16:01:45.879561+0000","last_active":"2024-09-17T16:01:45.944274+0000","last_peered":"2024-09-17T16:01:45.944274+0000","last_clean":"2024-09-17T16:01:45.944274+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:01:45.944274+0000","last_undegraded":"2024-09-17T16:01:45.944274+0000","last_fullsized":"2024-09-17T16:01:45.944274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.856272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95719047499999999,"stat_sum":{"num_bytes":3457024,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1376,"num_write":19,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.c","version":"44'12","reported_seq":67,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921602+0000","last_change":"2024-09-17T16:01:37.878565+0000","last_active":"2024-09-17T16:01:43.921602+0000","last_peered":"2024-09-17T16:01:43.921602+0000","last_clean":"2024-09-17T16:01:43.921602+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:01:43.921602+0000","last_undegraded":"2024-09-17T16:01:43.921602+0000","last_fullsized":"2024-09-17T16:01:43.921602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.083328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000730739,"stat_sum":{"num_bytes":1933312,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":9,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.b","version":"45'12","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921162+0000","last_change":"2024-09-17T16:01:37.890455+0000","last_active":"2024-09-17T16:01:43.921162+0000","last_peered":"2024-09-17T16:01:43.921162+0000","last_clean":"2024-09-17T16:01:43.921162+0000","last_became_active":"2024-09-17T16:01:32.027225+0000","last_became_peered":"2024-09-17T16:01:32.027225+0000","last_unstale":"2024-09-17T16:01:43.921162+0000","last_undegraded":"2024-09-17T16:01:43.921162+0000","last_fullsized":"2024-09-17T16:01:43.921162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:35.171662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080637500000000002,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":816,"num_write":9,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,7],"acting":[13,1,11,15,6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.a","version":"47'20","reported_seq":85,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944277+0000","last_change":"2024-09-17T16:01:44.938624+0000","last_active":"2024-09-17T16:01:45.944277+0000","last_peered":"2024-09-17T16:01:45.944277+0000","last_clean":"2024-09-17T16:01:45.944277+0000","last_became_active":"2024-09-17T16:01:32.579608+0000","last_became_peered":"2024-09-17T16:01:32.579608+0000","last_unstale":"2024-09-17T16:01:45.944277+0000","last_undegraded":"2024-09-17T16:01:45.944277+0000","last_fullsized":"2024-09-17T16:01:45.944277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:57.246615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016324966999999999,"stat_sum":{"num_bytes":3293184,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,6,8],"acting":[12,4,7,3,14,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.9","version":"44'15","reported_seq":74,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947381+0000","last_change":"2024-09-17T16:01:37.875257+0000","last_active":"2024-09-17T16:01:43.947381+0000","last_peered":"2024-09-17T16:01:43.947381+0000","last_clean":"2024-09-17T16:01:43.947381+0000","last_became_active":"2024-09-17T16:01:32.027179+0000","last_became_peered":"2024-09-17T16:01:32.027179+0000","last_unstale":"2024-09-17T16:01:43.947381+0000","last_undegraded":"2024-09-17T16:01:43.947381+0000","last_fullsized":"2024-09-17T16:01:43.947381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.899899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085827199999999998,"stat_sum":{"num_bytes":4128768,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":13,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.8","version":"44'25","reported_seq":81,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921816+0000","last_change":"2024-09-17T16:01:37.884652+0000","last_active":"2024-09-17T16:01:43.921816+0000","last_peered":"2024-09-17T16:01:43.921816+0000","last_clean":"2024-09-17T16:01:43.921816+0000","last_became_active":"2024-09-17T16:01:31.848773+0000","last_became_peered":"2024-09-17T16:01:31.848773+0000","last_unstale":"2024-09-17T16:01:43.921816+0000","last_undegraded":"2024-09-17T16:01:43.921816+0000","last_fullsized":"2024-09-17T16:01:43.921816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:55.083415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096304449999999996,"stat_sum":{"num_bytes":3784704,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":15,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":189,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921662+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:43.921662+0000","last_peered":"2024-09-17T16:01:43.921662+0000","last_clean":"2024-09-17T16:01:43.921662+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:43.921662+0000","last_undegraded":"2024-09-17T16:01:43.921662+0000","last_fullsized":"2024-09-17T16:01:43.921662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":55,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921551+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:43.921551+0000","last_peered":"2024-09-17T16:01:43.921551+0000","last_clean":"2024-09-17T16:01:43.921551+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:43.921551+0000","last_undegraded":"2024-09-17T16:01:43.921551+0000","last_fullsized":"2024-09-17T16:01:43.921551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'18","reported_seq":86,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.940205+0000","last_change":"2024-09-17T16:01:37.875712+0000","last_active":"2024-09-17T16:01:43.940205+0000","last_peered":"2024-09-17T16:01:43.940205+0000","last_clean":"2024-09-17T16:01:43.940205+0000","last_became_active":"2024-09-17T16:01:32.579280+0000","last_became_peered":"2024-09-17T16:01:32.579280+0000","last_unstale":"2024-09-17T16:01:43.940205+0000","last_undegraded":"2024-09-17T16:01:43.940205+0000","last_fullsized":"2024-09-17T16:01:43.940205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:40.033715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078007100000000002,"stat_sum":{"num_bytes":3112960,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":22,"num_read_kb":2062,"num_write":14,"num_write_kb":2960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929150+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:43.929150+0000","last_peered":"2024-09-17T16:01:43.929150+0000","last_clean":"2024-09-17T16:01:43.929150+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:43.929150+0000","last_undegraded":"2024-09-17T16:01:43.929150+0000","last_fullsized":"2024-09-17T16:01:43.929150+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"35'6","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.920904+0000","last_change":"2024-09-17T16:01:37.890929+0000","last_active":"2024-09-17T16:01:43.920904+0000","last_peered":"2024-09-17T16:01:43.920904+0000","last_clean":"2024-09-17T16:01:43.920904+0000","last_became_active":"2024-09-17T16:01:32.579478+0000","last_became_peered":"2024-09-17T16:01:32.579478+0000","last_unstale":"2024-09-17T16:01:43.920904+0000","last_undegraded":"2024-09-17T16:01:43.920904+0000","last_fullsized":"2024-09-17T16:01:43.920904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.792345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091054000000000005,"stat_sum":{"num_bytes":1245184,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.930873+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:43.930873+0000","last_peered":"2024-09-17T16:01:43.930873+0000","last_clean":"2024-09-17T16:01:43.930873+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:43.930873+0000","last_undegraded":"2024-09-17T16:01:43.930873+0000","last_fullsized":"2024-09-17T16:01:43.930873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"44'15","reported_seq":75,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947349+0000","last_change":"2024-09-17T16:01:37.874906+0000","last_active":"2024-09-17T16:01:43.947349+0000","last_peered":"2024-09-17T16:01:43.947349+0000","last_clean":"2024-09-17T16:01:43.947349+0000","last_became_active":"2024-09-17T16:01:32.578748+0000","last_became_peered":"2024-09-17T16:01:32.578748+0000","last_unstale":"2024-09-17T16:01:43.947349+0000","last_undegraded":"2024-09-17T16:01:43.947349+0000","last_fullsized":"2024-09-17T16:01:43.947349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:31.899666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051024999999999996,"stat_sum":{"num_bytes":2113536,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":135,"num_write":31,"num_write_kb":2064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":56,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929283+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:43.929283+0000","last_peered":"2024-09-17T16:01:43.929283+0000","last_clean":"2024-09-17T16:01:43.929283+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:43.929283+0000","last_undegraded":"2024-09-17T16:01:43.929283+0000","last_fullsized":"2024-09-17T16:01:43.929283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"39'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931043+0000","last_change":"2024-09-17T16:01:37.886987+0000","last_active":"2024-09-17T16:01:43.931043+0000","last_peered":"2024-09-17T16:01:43.931043+0000","last_clean":"2024-09-17T16:01:43.931043+0000","last_became_active":"2024-09-17T16:01:32.027376+0000","last_became_peered":"2024-09-17T16:01:32.027376+0000","last_unstale":"2024-09-17T16:01:43.931043+0000","last_undegraded":"2024-09-17T16:01:43.931043+0000","last_fullsized":"2024-09-17T16:01:43.931043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.273009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069738480000000004,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.5","version":"35'4","reported_seq":61,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929570+0000","last_change":"2024-09-17T16:01:37.885001+0000","last_active":"2024-09-17T16:01:43.929570+0000","last_peered":"2024-09-17T16:01:43.929570+0000","last_clean":"2024-09-17T16:01:43.929570+0000","last_became_active":"2024-09-17T16:01:31.981156+0000","last_became_peered":"2024-09-17T16:01:31.981156+0000","last_unstale":"2024-09-17T16:01:43.929570+0000","last_undegraded":"2024-09-17T16:01:43.929570+0000","last_fullsized":"2024-09-17T16:01:43.929570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.852744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066925399999999999,"stat_sum":{"num_bytes":1130496,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-17T16:01:48.337155+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:48.337155+0000","last_peered":"2024-09-17T16:01:48.337155+0000","last_clean":"2024-09-17T16:01:48.337155+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:48.337155+0000","last_undegraded":"2024-09-17T16:01:48.337155+0000","last_fullsized":"2024-09-17T16:01:48.337155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":97,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-17T16:01:48.341887+0000","last_change":"2024-09-17T16:01:46.937570+0000","last_active":"2024-09-17T16:01:48.341887+0000","last_peered":"2024-09-17T16:01:48.341887+0000","last_clean":"2024-09-17T16:01:48.341887+0000","last_became_active":"2024-09-17T16:01:32.579546+0000","last_became_peered":"2024-09-17T16:01:32.579546+0000","last_unstale":"2024-09-17T16:01:48.341887+0000","last_undegraded":"2024-09-17T16:01:48.341887+0000","last_fullsized":"2024-09-17T16:01:48.341887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.890503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082056999999999998,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":915,"num_write":8,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.906323+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:43.906323+0000","last_peered":"2024-09-17T16:01:43.906323+0000","last_clean":"2024-09-17T16:01:43.906323+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:43.906323+0000","last_undegraded":"2024-09-17T16:01:43.906323+0000","last_fullsized":"2024-09-17T16:01:43.906323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"42'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921053+0000","last_change":"2024-09-17T16:01:37.890423+0000","last_active":"2024-09-17T16:01:43.921053+0000","last_peered":"2024-09-17T16:01:43.921053+0000","last_clean":"2024-09-17T16:01:43.921053+0000","last_became_active":"2024-09-17T16:01:31.981906+0000","last_became_peered":"2024-09-17T16:01:31.981906+0000","last_unstale":"2024-09-17T16:01:43.921053+0000","last_undegraded":"2024-09-17T16:01:43.921053+0000","last_fullsized":"2024-09-17T16:01:43.921053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.018677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073874200000000002,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.928508+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:43.928508+0000","last_peered":"2024-09-17T16:01:43.928508+0000","last_clean":"2024-09-17T16:01:43.928508+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:43.928508+0000","last_undegraded":"2024-09-17T16:01:43.928508+0000","last_fullsized":"2024-09-17T16:01:43.928508+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"35'6","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921133+0000","last_change":"2024-09-17T16:01:37.890930+0000","last_active":"2024-09-17T16:01:43.921133+0000","last_peered":"2024-09-17T16:01:43.921133+0000","last_clean":"2024-09-17T16:01:43.921133+0000","last_became_active":"2024-09-17T16:01:31.981275+0000","last_became_peered":"2024-09-17T16:01:31.981275+0000","last_unstale":"2024-09-17T16:01:43.921133+0000","last_undegraded":"2024-09-17T16:01:43.921133+0000","last_fullsized":"2024-09-17T16:01:43.921133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:51.368632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089277400000000004,"stat_sum":{"num_bytes":1114112,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":1815,"num_write":6,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929359+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:43.929359+0000","last_peered":"2024-09-17T16:01:43.929359+0000","last_clean":"2024-09-17T16:01:43.929359+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:43.929359+0000","last_undegraded":"2024-09-17T16:01:43.929359+0000","last_fullsized":"2024-09-17T16:01:43.929359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37158912,"num_objects":76,"num_object_clones":28,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":1,"num_read":169,"num_read_kb":9969,"num_write":203,"num_write_kb":35616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61644800,"data_stored":79712256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247818,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30240,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341600,"statfs":{"total":96636764160,"available":96605798400,"internally_reserved":0,"allocated":3284992,"data_stored":4304096,"data_compressed":75108,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30652,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341188,"statfs":{"total":96636764160,"available":96605376512,"internally_reserved":0,"allocated":3706880,"data_stored":4689046,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":4320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":4423680,"data_stored":5528726,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35176,"kb_used_data":3980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336664,"statfs":{"total":96636764160,"available":96600743936,"internally_reserved":0,"allocated":4075520,"data_stored":5596571,"data_compressed":84232,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":5705728,"data_stored":6978710,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":5533696,"data_stored":6651030,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":4780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":4894720,"data_stored":6549174,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30124,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341716,"statfs":{"total":96636764160,"available":96605917184,"internally_reserved":0,"allocated":3166208,"data_stored":3673238,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":6164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":6311936,"data_stored":7478422,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30084,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341756,"statfs":{"total":96636764160,"available":96605958144,"internally_reserved":0,"allocated":3125248,"data_stored":3433613,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":4536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":4644864,"data_stored":6184086,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30728,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341112,"statfs":{"total":96636764160,"available":96605298688,"internally_reserved":0,"allocated":3784704,"data_stored":4463766,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30760,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341080,"statfs":{"total":96636764160,"available":96605265920,"internally_reserved":0,"allocated":3817472,"data_stored":4193449,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30020,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341820,"statfs":{"total":96636764160,"available":96606023680,"internally_reserved":0,"allocated":3063808,"data_stored":3232928,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29776,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342064,"statfs":{"total":96636764160,"available":96606273536,"internally_reserved":0,"allocated":2805760,"data_stored":2903190,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33660,"kb_used_data":6632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338180,"statfs":{"total":96636764160,"available":96602296320,"internally_reserved":0,"allocated":6791168,"data_stored":8998582,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5226496,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3624960,"data_stored":5341184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2686976,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5832704,"data_stored":7184384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":5890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3305472,"data_stored":4169728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3334144,"data_stored":3899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2592768,"data_stored":3055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2326528,"data_stored":2609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6254592,"data_stored":8175616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2781184,"data_stored":3989504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5054464,"data_stored":6356992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3944448,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3227648,"data_stored":4395008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:49.008 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-17T16:01:49.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:01:49.009 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-17T16:01:49.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:49.056 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:49.053+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.157 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.158 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:49.156+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:49.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:01:49.258 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:49.256+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:49.277 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.277 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:01:49.339 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:123: done (3 left) 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:124: done (2 left) 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:125: done (1 left) 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:126: done (0 left) 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 13 2024-09-17T16:01:49.348 INFO:tasks.rados.rados.0.smithi181.stdout:127: seq_num 71 ranges {2015232=425984,2441216=573440,3014656=196608} 2024-09-17T16:01:49.356 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:49.356 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:49.359 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:49.357+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:49.360 INFO:tasks.rados.rados.0.smithi181.stdout:127: writing smithi18136759-50 from 2015232 to 2441216 tid 1 2024-09-17T16:01:49.363 INFO:tasks.rados.rados.0.smithi181.stdout:127: writing smithi18136759-50 from 2441216 to 3014656 tid 2 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:127: writing smithi18136759-50 from 3014656 to 3211264 tid 3 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:128: read oid 26 snap -1 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:128: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:129: rollback oid 8 current snap is 13 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 8 to 11 2024-09-17T16:01:49.365 INFO:tasks.rados.rados.0.smithi181.stdout:130: snap_remove snap 9 2024-09-17T16:01:49.371 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-17T16:01:49.339227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37749299,"num_objects":80,"num_object_clones":28,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":1,"num_read":255,"num_read_kb":10042,"num_write":350,"num_write_kb":38290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":333,"ondisk_log_size":333,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":508356,"kb_used_data":67516,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509441084,"statfs":{"total":1546188226560,"available":1545667670016,"internally_reserved":0,"allocated":69136384,"data_stored":84858627,"data_compressed":1140662,"data_compressed_allocated":1601536,"data_compressed_original":4079616,"omap_allocated":25519,"internal_metadata":450993233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2129920,"num_objects":4,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":2283,"num_write":10,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.424004"},"pg_stats":[{"pgid":"3.f","version":"44'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921616+0000","last_change":"2024-09-17T16:01:37.883270+0000","last_active":"2024-09-17T16:01:43.921616+0000","last_peered":"2024-09-17T16:01:43.921616+0000","last_clean":"2024-09-17T16:01:43.921616+0000","last_became_active":"2024-09-17T16:01:31.980473+0000","last_became_peered":"2024-09-17T16:01:31.980473+0000","last_unstale":"2024-09-17T16:01:43.921616+0000","last_undegraded":"2024-09-17T16:01:43.921616+0000","last_fullsized":"2024-09-17T16:01:43.921616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:53.770253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052621489999999998,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":367,"num_write":8,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.e","version":"45'16","reported_seq":79,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931984+0000","last_change":"2024-09-17T16:01:37.870258+0000","last_active":"2024-09-17T16:01:43.931984+0000","last_peered":"2024-09-17T16:01:43.931984+0000","last_clean":"2024-09-17T16:01:43.931984+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:01:43.931984+0000","last_undegraded":"2024-09-17T16:01:43.931984+0000","last_fullsized":"2024-09-17T16:01:43.931984+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.879288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045074600000000002,"stat_sum":{"num_bytes":3620864,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":13,"num_read_kb":2454,"num_write":33,"num_write_kb":3536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.d","version":"47'23","reported_seq":100,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944274+0000","last_change":"2024-09-17T16:01:45.879561+0000","last_active":"2024-09-17T16:01:45.944274+0000","last_peered":"2024-09-17T16:01:45.944274+0000","last_clean":"2024-09-17T16:01:45.944274+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:01:45.944274+0000","last_undegraded":"2024-09-17T16:01:45.944274+0000","last_fullsized":"2024-09-17T16:01:45.944274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.856272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.95719047499999999,"stat_sum":{"num_bytes":3457024,"num_objects":7,"num_object_clones":2,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1376,"num_write":19,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.c","version":"44'12","reported_seq":67,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921602+0000","last_change":"2024-09-17T16:01:37.878565+0000","last_active":"2024-09-17T16:01:43.921602+0000","last_peered":"2024-09-17T16:01:43.921602+0000","last_clean":"2024-09-17T16:01:43.921602+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:01:43.921602+0000","last_undegraded":"2024-09-17T16:01:43.921602+0000","last_fullsized":"2024-09-17T16:01:43.921602+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.083328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000730739,"stat_sum":{"num_bytes":1933312,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":9,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.b","version":"45'12","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921162+0000","last_change":"2024-09-17T16:01:37.890455+0000","last_active":"2024-09-17T16:01:43.921162+0000","last_peered":"2024-09-17T16:01:43.921162+0000","last_clean":"2024-09-17T16:01:43.921162+0000","last_became_active":"2024-09-17T16:01:32.027225+0000","last_became_peered":"2024-09-17T16:01:32.027225+0000","last_unstale":"2024-09-17T16:01:43.921162+0000","last_undegraded":"2024-09-17T16:01:43.921162+0000","last_fullsized":"2024-09-17T16:01:43.921162+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:35.171662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080637500000000002,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":816,"num_write":9,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,7],"acting":[13,1,11,15,6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.a","version":"47'20","reported_seq":85,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-17T16:01:45.944277+0000","last_change":"2024-09-17T16:01:44.938624+0000","last_active":"2024-09-17T16:01:45.944277+0000","last_peered":"2024-09-17T16:01:45.944277+0000","last_clean":"2024-09-17T16:01:45.944277+0000","last_became_active":"2024-09-17T16:01:32.579608+0000","last_became_peered":"2024-09-17T16:01:32.579608+0000","last_unstale":"2024-09-17T16:01:45.944277+0000","last_undegraded":"2024-09-17T16:01:45.944277+0000","last_fullsized":"2024-09-17T16:01:45.944277+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:57.246615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016324966999999999,"stat_sum":{"num_bytes":3293184,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":12,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,6,8],"acting":[12,4,7,3,14,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"2"}]},{"pgid":"3.9","version":"44'15","reported_seq":74,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947381+0000","last_change":"2024-09-17T16:01:37.875257+0000","last_active":"2024-09-17T16:01:43.947381+0000","last_peered":"2024-09-17T16:01:43.947381+0000","last_clean":"2024-09-17T16:01:43.947381+0000","last_became_active":"2024-09-17T16:01:32.027179+0000","last_became_peered":"2024-09-17T16:01:32.027179+0000","last_unstale":"2024-09-17T16:01:43.947381+0000","last_undegraded":"2024-09-17T16:01:43.947381+0000","last_fullsized":"2024-09-17T16:01:43.947381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.899899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085827199999999998,"stat_sum":{"num_bytes":4128768,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":13,"num_write_kb":4032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.8","version":"44'25","reported_seq":81,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921816+0000","last_change":"2024-09-17T16:01:37.884652+0000","last_active":"2024-09-17T16:01:43.921816+0000","last_peered":"2024-09-17T16:01:43.921816+0000","last_clean":"2024-09-17T16:01:43.921816+0000","last_became_active":"2024-09-17T16:01:31.848773+0000","last_became_peered":"2024-09-17T16:01:31.848773+0000","last_unstale":"2024-09-17T16:01:43.921816+0000","last_undegraded":"2024-09-17T16:01:43.921816+0000","last_fullsized":"2024-09-17T16:01:43.921816+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:55.083415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096304449999999996,"stat_sum":{"num_bytes":3784704,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":15,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":189,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921662+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:43.921662+0000","last_peered":"2024-09-17T16:01:43.921662+0000","last_clean":"2024-09-17T16:01:43.921662+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:43.921662+0000","last_undegraded":"2024-09-17T16:01:43.921662+0000","last_fullsized":"2024-09-17T16:01:43.921662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":55,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921551+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:43.921551+0000","last_peered":"2024-09-17T16:01:43.921551+0000","last_clean":"2024-09-17T16:01:43.921551+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:43.921551+0000","last_undegraded":"2024-09-17T16:01:43.921551+0000","last_fullsized":"2024-09-17T16:01:43.921551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"45'18","reported_seq":86,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.940205+0000","last_change":"2024-09-17T16:01:37.875712+0000","last_active":"2024-09-17T16:01:43.940205+0000","last_peered":"2024-09-17T16:01:43.940205+0000","last_clean":"2024-09-17T16:01:43.940205+0000","last_became_active":"2024-09-17T16:01:32.579280+0000","last_became_peered":"2024-09-17T16:01:32.579280+0000","last_unstale":"2024-09-17T16:01:43.940205+0000","last_undegraded":"2024-09-17T16:01:43.940205+0000","last_fullsized":"2024-09-17T16:01:43.940205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:40.033715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078007100000000002,"stat_sum":{"num_bytes":3112960,"num_objects":7,"num_object_clones":3,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":22,"num_read_kb":2062,"num_write":14,"num_write_kb":2960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929150+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:43.929150+0000","last_peered":"2024-09-17T16:01:43.929150+0000","last_clean":"2024-09-17T16:01:43.929150+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:43.929150+0000","last_undegraded":"2024-09-17T16:01:43.929150+0000","last_fullsized":"2024-09-17T16:01:43.929150+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"35'6","reported_seq":64,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.920904+0000","last_change":"2024-09-17T16:01:37.890929+0000","last_active":"2024-09-17T16:01:43.920904+0000","last_peered":"2024-09-17T16:01:43.920904+0000","last_clean":"2024-09-17T16:01:43.920904+0000","last_became_active":"2024-09-17T16:01:32.579478+0000","last_became_peered":"2024-09-17T16:01:32.579478+0000","last_unstale":"2024-09-17T16:01:43.920904+0000","last_undegraded":"2024-09-17T16:01:43.920904+0000","last_fullsized":"2024-09-17T16:01:43.920904+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.792345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091054000000000005,"stat_sum":{"num_bytes":1245184,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.930873+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:43.930873+0000","last_peered":"2024-09-17T16:01:43.930873+0000","last_clean":"2024-09-17T16:01:43.930873+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:43.930873+0000","last_undegraded":"2024-09-17T16:01:43.930873+0000","last_fullsized":"2024-09-17T16:01:43.930873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"44'15","reported_seq":75,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.947349+0000","last_change":"2024-09-17T16:01:37.874906+0000","last_active":"2024-09-17T16:01:43.947349+0000","last_peered":"2024-09-17T16:01:43.947349+0000","last_clean":"2024-09-17T16:01:43.947349+0000","last_became_active":"2024-09-17T16:01:32.578748+0000","last_became_peered":"2024-09-17T16:01:32.578748+0000","last_unstale":"2024-09-17T16:01:43.947349+0000","last_undegraded":"2024-09-17T16:01:43.947349+0000","last_fullsized":"2024-09-17T16:01:43.947349+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:31.899666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051024999999999996,"stat_sum":{"num_bytes":2113536,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":135,"num_write":31,"num_write_kb":2064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":56,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929283+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:43.929283+0000","last_peered":"2024-09-17T16:01:43.929283+0000","last_clean":"2024-09-17T16:01:43.929283+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:43.929283+0000","last_undegraded":"2024-09-17T16:01:43.929283+0000","last_fullsized":"2024-09-17T16:01:43.929283+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"39'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.931043+0000","last_change":"2024-09-17T16:01:37.886987+0000","last_active":"2024-09-17T16:01:43.931043+0000","last_peered":"2024-09-17T16:01:43.931043+0000","last_clean":"2024-09-17T16:01:43.931043+0000","last_became_active":"2024-09-17T16:01:32.027376+0000","last_became_peered":"2024-09-17T16:01:32.027376+0000","last_unstale":"2024-09-17T16:01:43.931043+0000","last_undegraded":"2024-09-17T16:01:43.931043+0000","last_fullsized":"2024-09-17T16:01:43.931043+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.273009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069738480000000004,"stat_sum":{"num_bytes":2441216,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":2384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"3.5","version":"35'4","reported_seq":61,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929570+0000","last_change":"2024-09-17T16:01:37.885001+0000","last_active":"2024-09-17T16:01:43.929570+0000","last_peered":"2024-09-17T16:01:43.929570+0000","last_clean":"2024-09-17T16:01:43.929570+0000","last_became_active":"2024-09-17T16:01:31.981156+0000","last_became_peered":"2024-09-17T16:01:31.981156+0000","last_unstale":"2024-09-17T16:01:43.929570+0000","last_undegraded":"2024-09-17T16:01:43.929570+0000","last_fullsized":"2024-09-17T16:01:43.929570+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.852744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066925399999999999,"stat_sum":{"num_bytes":1130496,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-17T16:01:48.337155+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:48.337155+0000","last_peered":"2024-09-17T16:01:48.337155+0000","last_clean":"2024-09-17T16:01:48.337155+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:48.337155+0000","last_undegraded":"2024-09-17T16:01:48.337155+0000","last_fullsized":"2024-09-17T16:01:48.337155+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":97,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-17T16:01:48.341887+0000","last_change":"2024-09-17T16:01:46.937570+0000","last_active":"2024-09-17T16:01:48.341887+0000","last_peered":"2024-09-17T16:01:48.341887+0000","last_clean":"2024-09-17T16:01:48.341887+0000","last_became_active":"2024-09-17T16:01:32.579546+0000","last_became_peered":"2024-09-17T16:01:32.579546+0000","last_unstale":"2024-09-17T16:01:48.341887+0000","last_undegraded":"2024-09-17T16:01:48.341887+0000","last_fullsized":"2024-09-17T16:01:48.341887+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.890503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082056999999999998,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":915,"num_write":8,"num_write_kb":1040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.906323+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:43.906323+0000","last_peered":"2024-09-17T16:01:43.906323+0000","last_clean":"2024-09-17T16:01:43.906323+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:43.906323+0000","last_undegraded":"2024-09-17T16:01:43.906323+0000","last_fullsized":"2024-09-17T16:01:43.906323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.7","version":"42'10","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921053+0000","last_change":"2024-09-17T16:01:37.890423+0000","last_active":"2024-09-17T16:01:43.921053+0000","last_peered":"2024-09-17T16:01:43.921053+0000","last_clean":"2024-09-17T16:01:43.921053+0000","last_became_active":"2024-09-17T16:01:31.981906+0000","last_became_peered":"2024-09-17T16:01:31.981906+0000","last_unstale":"2024-09-17T16:01:43.921053+0000","last_undegraded":"2024-09-17T16:01:43.921053+0000","last_fullsized":"2024-09-17T16:01:43.921053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.018677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073874200000000002,"stat_sum":{"num_bytes":2523136,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":9,"num_write_kb":2464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.928508+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:43.928508+0000","last_peered":"2024-09-17T16:01:43.928508+0000","last_clean":"2024-09-17T16:01:43.928508+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:43.928508+0000","last_undegraded":"2024-09-17T16:01:43.928508+0000","last_fullsized":"2024-09-17T16:01:43.928508+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.6","version":"35'6","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.921133+0000","last_change":"2024-09-17T16:01:37.890930+0000","last_active":"2024-09-17T16:01:43.921133+0000","last_peered":"2024-09-17T16:01:43.921133+0000","last_clean":"2024-09-17T16:01:43.921133+0000","last_became_active":"2024-09-17T16:01:31.981275+0000","last_became_peered":"2024-09-17T16:01:31.981275+0000","last_unstale":"2024-09-17T16:01:43.921133+0000","last_undegraded":"2024-09-17T16:01:43.921133+0000","last_fullsized":"2024-09-17T16:01:43.921133+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:51.368632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089277400000000004,"stat_sum":{"num_bytes":1114112,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":1815,"num_write":6,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":54,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T16:01:43.929359+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:43.929359+0000","last_peered":"2024-09-17T16:01:43.929359+0000","last_clean":"2024-09-17T16:01:43.929359+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:43.929359+0000","last_undegraded":"2024-09-17T16:01:43.929359+0000","last_fullsized":"2024-09-17T16:01:43.929359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":37158912,"num_objects":76,"num_object_clones":28,"num_object_copies":532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":1,"num_read":169,"num_read_kb":9969,"num_write":203,"num_write_kb":35616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61644800,"data_stored":79712256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247818,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30240,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341600,"statfs":{"total":96636764160,"available":96605798400,"internally_reserved":0,"allocated":3284992,"data_stored":4304096,"data_compressed":75108,"data_compressed_allocated":102400,"data_compressed_original":204800,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30652,"kb_used_data":3620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341188,"statfs":{"total":96636764160,"available":96605376512,"internally_reserved":0,"allocated":3706880,"data_stored":4689046,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":4320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":4423680,"data_stored":5528726,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35176,"kb_used_data":3980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336664,"statfs":{"total":96636764160,"available":96600743936,"internally_reserved":0,"allocated":4075520,"data_stored":5596571,"data_compressed":84232,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":5705728,"data_stored":6978710,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36600,"kb_used_data":5404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335240,"statfs":{"total":96636764160,"available":96599285760,"internally_reserved":0,"allocated":5533696,"data_stored":6651030,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31808,"kb_used_data":4780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340032,"statfs":{"total":96636764160,"available":96604192768,"internally_reserved":0,"allocated":4894720,"data_stored":6549174,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30124,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341716,"statfs":{"total":96636764160,"available":96605917184,"internally_reserved":0,"allocated":3166208,"data_stored":3673238,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33200,"kb_used_data":6164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338640,"statfs":{"total":96636764160,"available":96602767360,"internally_reserved":0,"allocated":6311936,"data_stored":7478422,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30084,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341756,"statfs":{"total":96636764160,"available":96605958144,"internally_reserved":0,"allocated":3125248,"data_stored":3433613,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":4536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":4644864,"data_stored":6184086,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30728,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341112,"statfs":{"total":96636764160,"available":96605298688,"internally_reserved":0,"allocated":3784704,"data_stored":4463766,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30760,"kb_used_data":3728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341080,"statfs":{"total":96636764160,"available":96605265920,"internally_reserved":0,"allocated":3817472,"data_stored":4193449,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30020,"kb_used_data":2992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341820,"statfs":{"total":96636764160,"available":96606023680,"internally_reserved":0,"allocated":3063808,"data_stored":3232928,"data_compressed":65994,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29776,"kb_used_data":2740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342064,"statfs":{"total":96636764160,"available":96606273536,"internally_reserved":0,"allocated":2805760,"data_stored":2903190,"data_compressed":69028,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280520,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33660,"kb_used_data":6632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338180,"statfs":{"total":96636764160,"available":96602296320,"internally_reserved":0,"allocated":6791168,"data_stored":8998582,"data_compressed":79527,"data_compressed_allocated":131072,"data_compressed_original":700416,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5226496,"data_stored":6684672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3624960,"data_stored":5341184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4358144,"data_stored":5726208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2686976,"data_stored":3379200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5832704,"data_stored":7184384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2928640,"data_stored":3620864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4165632,"data_stored":5890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3305472,"data_stored":4169728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":3334144,"data_stored":3899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2592768,"data_stored":3055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2326528,"data_stored":2609152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":6254592,"data_stored":8175616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2781184,"data_stored":3989504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":5054464,"data_stored":6356992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":3944448,"data_stored":5234688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3227648,"data_stored":4395008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:49.373 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2024-09-17T16:01:49.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2024-09-17T16:01:49.402 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.402 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:01:49.495 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:49.493+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:49.561 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.562 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.562 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:49.559+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:49.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:01:49.662 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:49.659+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:49.719 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.719 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:01:49.763 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:49.761+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:49.848 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.849 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:01:49.863 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:49.862+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:49.964 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:49.962+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:49.965 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:49.965 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:49.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:01:49.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:01:50.065 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:50.062+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:50.081 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:50.081 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:50.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:01:50.114 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:50.114 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:50.136 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:50.136 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:50.158 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:50.158 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:50.166 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:50.164+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:50.180 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:50.180 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:50.197 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:50.197 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:50.200 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:50.200 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:50.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:01:50.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:50.222 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:50.244 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:50.244 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:50.265 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:50.265 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:50.266 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:50.264+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:50.285 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:50.286 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:50.307 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:50.313 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:50.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:50.313 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:50.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:01:50.329 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:50.329 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:50.345 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 3 pg_num to 26 2024-09-17T16:01:50.351 INFO:tasks.rados.rados.0.smithi181.stdout:130: done (3 left) 2024-09-17T16:01:50.351 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 34 current snap is 13 2024-09-17T16:01:50.351 INFO:tasks.rados.rados.0.smithi181.stdout:127: finishing write tid 1 to smithi18136759-50 2024-09-17T16:01:50.351 INFO:tasks.rados.rados.0.smithi181.stdout:131: seq_num 72 ranges {557056=442368,999424=720896,1720320=49152} 2024-09-17T16:01:50.352 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:50.352 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:50.356 INFO:tasks.rados.rados.0.smithi181.stdout:131: writing smithi18136759-34 from 557056 to 999424 tid 1 2024-09-17T16:01:50.361 INFO:tasks.rados.rados.0.smithi181.stdout:131: writing smithi18136759-34 from 999424 to 1720320 tid 2 2024-09-17T16:01:50.361 INFO:tasks.rados.rados.0.smithi181.stdout:131: writing smithi18136759-34 from 1720320 to 1769472 tid 3 2024-09-17T16:01:50.361 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 30 current snap is 13 2024-09-17T16:01:50.361 INFO:tasks.rados.rados.0.smithi181.stdout:132: seq_num 73 ranges {491520=458752,950272=475136,1425408=294912} 2024-09-17T16:01:50.367 INFO:tasks.rados.rados.0.smithi181.stdout:132: writing smithi18136759-30 from 491520 to 950272 tid 1 2024-09-17T16:01:50.369 INFO:tasks.rados.rados.0.smithi181.stdout:132: writing smithi18136759-30 from 950272 to 1425408 tid 2 2024-09-17T16:01:50.372 INFO:tasks.rados.rados.0.smithi181.stdout:132: writing smithi18136759-30 from 1425408 to 1720320 tid 3 2024-09-17T16:01:50.372 INFO:tasks.rados.rados.0.smithi181.stdout:133: snap_create 2024-09-17T16:01:50.372 INFO:tasks.rados.rados.0.smithi181.stdout:127: finishing write tid 2 to smithi18136759-50 2024-09-17T16:01:50.372 INFO:tasks.rados.rados.0.smithi181.stdout:128: expect (ObjNum 25 snap 0 seq_num 25) 2024-09-17T16:01:50.372 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 4648 2024-09-17T16:01:50.377 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:50.377 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:50.377 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:50.378 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:50.375+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:50.381 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:127: finishing write tid 3 to smithi18136759-50 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:127: finishing write tid 4 to smithi18136759-50 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:129: finishing rollback tid 0 to smithi18136759-8 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:129: finishing rollback tid 1 to smithi18136759-8 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:129: finishing rollback tid 2 to smithi18136759-8 2024-09-17T16:01:50.386 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 35 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: finishing write tid 5 to smithi18136759-50 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: oid 50 updating version 0 to 29 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: oid 50 updating version 29 to 31 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: oid 50 version 31 is already newer than 30 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: oid 50 version 31 is already newer than 28 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 31 (ObjNum 71 snap 13 seq_num 71) dirty exists 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:127: left oid 50 (ObjNum 71 snap 13 seq_num 71) 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: finishing write tid 1 to smithi18136759-34 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: finishing write tid 2 to smithi18136759-34 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: finishing write tid 3 to smithi18136759-34 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: finishing write tid 4 to smithi18136759-34 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: finishing write tid 5 to smithi18136759-34 2024-09-17T16:01:50.387 INFO:tasks.rados.rados.0.smithi181.stdout:131: oid 34 updating version 0 to 9 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:131: oid 34 version 9 is already newer than 8 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:131: oid 34 updating version 9 to 10 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:131: oid 34 updating version 10 to 11 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 11 (ObjNum 72 snap 13 seq_num 72) dirty exists 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:131: left oid 34 (ObjNum 72 snap 13 seq_num 72) 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 1 to smithi18136759-30 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 2 to smithi18136759-30 2024-09-17T16:01:50.388 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 3 to smithi18136759-30 2024-09-17T16:01:50.400 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:50.400 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:50.421 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:50.421 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:50.422 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:50.420+0000 7f5e209e4640 0 -- 172.21.15.6:0/3370380555 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f5e1c224860 msgr2=0x7f5e1c226c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=31, reqid=client.4560.0:225, at_version=44'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5339775590000002, 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011247220000000001, 2024-09-17T16:01:50.440 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903502+0000", 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903503+0000", 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.23e-07 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903525+0000", 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2575999999999999e-05 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903562+0000", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6705000000000002e-05 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903571+0000", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.054e-06 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903639+0000", 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8058999999999993e-05 2024-09-17T16:01:50.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.903651+0000", 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2313000000000001e-05 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.904589+0000", 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00093754699999999995 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.904627+0000", 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8345e-05 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=33, reqid=client.4560.0:233, at_version=44'9, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5147463850000005, 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136534, 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922735+0000", 2024-09-17T16:01:50.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1340000000000001e-06 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922924+0000", 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018932500000000001 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922965+0000", 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0547000000000003e-05 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922974+0000", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4080000000000004e-06 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923042+0000", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7964000000000002e-05 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923056+0000", 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4018e-05 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924052+0000", 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000996432 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924099+0000", 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6511999999999998e-05 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:50.449 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5088456040000002, 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:01:50.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:01:50.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=35, reqid=client.4560.0:242, at_version=44'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5071421860000003, 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023531030000000001, 2024-09-17T16:01:50.454 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930339+0000", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1909999999999999e-06 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931084+0000", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074480300000000004 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931122+0000", 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8195000000000003e-05 2024-09-17T16:01:50.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931132+0000", 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931291+0000", 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015850099999999999 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931302+0000", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1487e-05 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932657+0000", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001355001 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932691+0000", 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3426000000000002e-05 2024-09-17T16:01:50.458 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=36, reqid=client.4560.0:243, at_version=44'13, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5057457420000002, 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098751700000000004, 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931734+0000", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931735+0000", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931752+0000", 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6582000000000001e-05 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931783+0000", 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1599e-05 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931792+0000", 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6079999999999996e-06 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931919+0000", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012768799999999999 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931928+0000", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5590000000000006e-06 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932695+0000", 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076658200000000003 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932722+0000", 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7197000000000001e-05 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=37, reqid=client.4560.0:236, at_version=44'15, trim_to=0'0, roll_forward_to=44'11))", 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.5013979160000002, 2024-09-17T16:01:50.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015795060000000001, 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:50.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936083+0000", 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.279e-06 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936265+0000", 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018191799999999999 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936310+0000", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5139000000000003e-05 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936321+0000", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0631e-05 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936421+0000", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9679000000000006e-05 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936430+0000", 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1379999999999996e-06 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937627+0000", 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011968930000000001 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937662+0000", 2024-09-17T16:01:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4829000000000001e-05 2024-09-17T16:01:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5181766129999996, 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026452619999999998, 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.14e-07 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919991+0000", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068760399999999997 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920025+0000", 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4292000000000003e-05 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920043+0000", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8028000000000001e-05 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920118+0000", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4583000000000001e-05 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920129+0000", 2024-09-17T16:01:50.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1105e-05 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921921+0000", 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017915590000000001 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921949+0000", 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7977e-05 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=26, reqid=client.4560.0:265, at_version=45'15, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.921189+0000", 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5162913580000001, 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001294012, 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921189+0000", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921189+0000", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921189+0000", 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:01:50.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921515+0000", 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032509900000000003 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921548+0000", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3760999999999997e-05 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921560+0000", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1627999999999999e-05 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921617+0000", 2024-09-17T16:01:50.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7011e-05 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921622+0000", 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4489999999999998e-06 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.922456+0000", 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083389900000000003 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.922483+0000", 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6335000000000001e-05 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 46/32 ECSubWrite(tid=68, reqid=osd.13.0:68, at_version=46'12, trim_to=0'0, roll_forward_to=42'10))", 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5203820920000002, 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015030169999999999, 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.479 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917127+0000", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9088999999999999e-05 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917562+0000", 2024-09-17T16:01:50.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043494999999999997 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917573+0000", 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0749e-05 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917670+0000", 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7249000000000004e-05 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917681+0000", 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1224e-05 2024-09-17T16:01:50.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918552+0000", 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087075200000000003 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918601+0000", 2024-09-17T16:01:50.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8752999999999999e-05 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 46/32 ECSubWrite(tid=30, reqid=osd.2.0:30, at_version=46'18, trim_to=0'0, roll_forward_to=45'16))", 2024-09-17T16:01:50.485 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5198545360000004, 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018538020000000001, 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917626+0000", 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9600000000000001e-07 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917649+0000", 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3079e-05 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917683+0000", 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4134999999999998e-05 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917691+0000", 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9840000000000001e-06 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917767+0000", 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6228000000000003e-05 2024-09-17T16:01:50.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917778+0000", 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0507e-05 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919440+0000", 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016621909999999999 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919479+0000", 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9481999999999999e-05 2024-09-17T16:01:50.492 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5188526539999998, 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019496909999999999, 2024-09-17T16:01:50.493 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918628+0000", 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3699999999999999e-07 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918653+0000", 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5582999999999999e-05 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918733+0000", 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0156999999999998e-05 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918742+0000", 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5580000000000001e-06 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918812+0000", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0308999999999994e-05 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918823+0000", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0370999999999999e-05 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920536+0000", 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001713525 2024-09-17T16:01:50.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920577+0000", 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0951000000000001e-05 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:50.499 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5179292200000001, 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013059860000000001, 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919552+0000", 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919575+0000", 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3499e-05 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919611+0000", 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5547999999999999e-05 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919620+0000", 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3659999999999998e-06 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919715+0000", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0769999999999997e-06 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920812+0000", 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001088049 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920857+0000", 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4922999999999997e-05 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 46/32 ECSubWrite(tid=69, reqid=osd.13.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:01:50.504 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.923292+0000", 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.51418754, 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075613000000000004, 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923292+0000", 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923292+0000", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923293+0000", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9900000000000009e-07 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923306+0000", 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2422000000000001e-05 2024-09-17T16:01:50.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923366+0000", 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0445999999999997e-05 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923375+0000", 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6049999999999998e-06 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923456+0000", 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1275999999999999e-05 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.923469+0000", 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2799000000000001e-05 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924014+0000", 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054517400000000005 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924049+0000", 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4409000000000002e-05 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5120937879999996, 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:01:50.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:01:50.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:01:50.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:01:50.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=50, reqid=osd.14.0:50, at_version=46'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.930136+0000", 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5073441360000004, 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081088999999999998, 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930136+0000", 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930136+0000", 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930137+0000", 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5899999999999995e-07 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930159+0000", 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2113e-05 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930180+0000", 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1036e-05 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930189+0000", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9199999999999993e-06 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930262+0000", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3109999999999996e-05 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930271+0000", 2024-09-17T16:01:50.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0710000000000001e-06 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930919+0000", 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00064830300000000003 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.930947+0000", 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7478e-05 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=51, reqid=osd.14.0:50, at_version=0'0, trim_to=0'0, roll_forward_to=46'21))", 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.079193+0000", 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.3582873290000004, 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092038199999999997, 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079193+0000", 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079193+0000", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079194+0000", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7630000000000001e-06 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079215+0000", 2024-09-17T16:01:50.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1016999999999999e-05 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079257+0000", 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1118999999999997e-05 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079284+0000", 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7878999999999999e-05 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079479+0000", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000194382 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.079507+0000", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8345999999999999e-05 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.080072+0000", 2024-09-17T16:01:50.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056442899999999997 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.080113+0000", 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1446999999999998e-05 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 46/32 ECSubWrite(tid=31, reqid=osd.2.0:30, at_version=0'0, trim_to=0'0, roll_forward_to=46'18))", 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.169436+0000", 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.2680437280000003, 2024-09-17T16:01:50.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000762924, 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169436+0000", 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169436+0000", 2024-09-17T16:01:50.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169437+0000", 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2500000000000004e-07 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169447+0000", 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169483+0000", 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5663000000000001e-05 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169489+0000", 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1999999999999999e-06 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169561+0000", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1755999999999995e-05 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.169569+0000", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3240000000000003e-06 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.170159+0000", 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058938999999999997 2024-09-17T16:01:50.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.170199+0000", 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0351e-05 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5052681720000001, 2024-09-17T16:01:50.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=58, reqid=client.4560.0:279, at_version=47'22, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.934651+0000", 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5028284989999996, 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011317580000000001, 2024-09-17T16:01:50.534 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934651+0000", 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934651+0000", 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934653+0000", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2079999999999999e-06 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934785+0000", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013248700000000001 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934859+0000", 2024-09-17T16:01:50.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3790999999999999e-05 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934876+0000", 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7024000000000002e-05 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934971+0000", 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4840000000000007e-05 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.934978+0000", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1799999999999999e-06 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935746+0000", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00076757500000000005 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935783+0000", 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7653e-05 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=59, reqid=client.4560.0:280, at_version=47'23, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.936559+0000", 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5009209280000002, 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00085627900000000005, 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936559+0000", 2024-09-17T16:01:50.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936559+0000", 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936560+0000", 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.09e-07 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936588+0000", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8410000000000001e-05 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936622+0000", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3540000000000001e-05 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936628+0000", 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3090000000000001e-06 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936687+0000", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9262999999999998e-05 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.936698+0000", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0757999999999999e-05 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.937379+0000", 2024-09-17T16:01:50.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068044899999999996 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.937415+0000", 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6740999999999999e-05 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.543 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:50.545 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:50.541+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:50.545 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 4 to smithi18136759-30 2024-09-17T16:01:50.545 INFO:tasks.rados.rados.0.smithi181.stdout:132: finishing write tid 5 to smithi18136759-30 2024-09-17T16:01:50.545 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 30 updating version 0 to 10 2024-09-17T16:01:50.545 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 30 updating version 10 to 11 2024-09-17T16:01:50.545 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 30 version 11 is already newer than 8 2024-09-17T16:01:50.546 INFO:tasks.rados.rados.0.smithi181.stdout:132: oid 30 version 11 is already newer than 9 2024-09-17T16:01:50.546 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 11 (ObjNum 73 snap 13 seq_num 73) dirty exists 2024-09-17T16:01:50.546 INFO:tasks.rados.rados.0.smithi181.stdout:132: left oid 30 (ObjNum 73 snap 13 seq_num 73) 2024-09-17T16:01:50.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:01:50.645 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:50.643+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:50.645 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:50.646 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:50.653 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:50.654 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:50.654 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:50.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:01:50.746 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:50.744+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:50.810 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:50.810 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:50.810 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:50.810 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:50.810 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:50.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:01:50.859 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:50.857+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:50.948 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:50.946+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:50.967 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:50.967 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:7, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.110810439000002, 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001460104, 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.968 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853946+0000", 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7230000000000001e-06 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854002+0000", 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677999999999998e-05 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854062+0000", 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-05 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854075+0000", 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2747e-05 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854189+0000", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114376 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854203+0000", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4100000000000001e-05 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855346+0000", 2024-09-17T16:01:50.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011423010000000001 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855405+0000", 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8949000000000001e-05 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:50.973 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.108427801000001, 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001864189, 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:50.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856328+0000", 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3570000000000001e-06 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856466+0000", 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137415 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856516+0000", 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0541999999999998e-05 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856527+0000", 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4474999999999994e-05 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856611+0000", 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0720000000000006e-06 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858143+0000", 2024-09-17T16:01:50.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015325499999999999 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858191+0000", 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000003e-05 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.107806363000002, 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.979 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:50.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:01:50.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.986 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.104843871, 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.987 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:01:50.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:01:50.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.101268747999999, 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001910307, 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863487+0000", 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-17T16:01:50.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863535+0000", 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7713999999999998e-05 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863653+0000", 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117757 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:50.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863662+0000", 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9779999999999994e-06 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863800+0000", 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013886599999999999 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863813+0000", 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.295e-05 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865357+0000", 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001543464 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865396+0000", 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9629000000000003e-05 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:50.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.101213541, 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:50.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=2, reqid=client.4560.0:61, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:51.003 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.080126716999999, 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018138760000000001, 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9e-08 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884974+0000", 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034541900000000002 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885013+0000", 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000002e-05 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885023+0000", 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885103+0000", 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9871999999999997e-05 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885112+0000", 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4769999999999992e-06 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886400+0000", 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012878869999999999 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886442+0000", 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2202e-05 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.008 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=11, reqid=client.4560.0:91, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.057042163999999, 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001672984, 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:51.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907717+0000", 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6290000000000001e-06 2024-09-17T16:01:51.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907950+0000", 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023220000000000001 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907988+0000", 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8900999999999999e-05 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907997+0000", 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908086+0000", 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908108+0000", 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2447e-05 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909346+0000", 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001237782 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909386+0000", 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9554000000000001e-05 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=13, reqid=client.4560.0:94, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.053524572000001, 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012298319999999999, 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911230+0000", 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911231+0000", 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1200000000000002e-07 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911496+0000", 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026446999999999999 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911523+0000", 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7665e-05 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911529+0000", 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1530000000000002e-06 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911584+0000", 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4611000000000003e-05 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.911592+0000", 2024-09-17T16:01:51.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912421+0000", 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082916399999999996 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.912460+0000", 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9047000000000001e-05 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=17, reqid=client.4560.0:124, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.016135836, 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016306890000000001, 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.020 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948620+0000", 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.238e-06 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949131+0000", 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051077299999999996 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949166+0000", 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4619999999999997e-05 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949174+0000", 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2400000000000007e-06 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949224+0000", 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0534000000000001e-05 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949231+0000", 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.229e-06 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950221+0000", 2024-09-17T16:01:51.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098999599999999993 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950250+0000", 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9059000000000001e-05 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.025 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.077052716000001, 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:01:51.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.072500756, 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.0380978169999997, 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019463340000000001, 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:51.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9200000000000002e-07 2024-09-17T16:01:51.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037e-05 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926741+0000", 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0134999999999999e-05 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926751+0000", 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926822+0000", 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0667000000000006e-05 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926854+0000", 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:01:51.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928578+0000", 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017237979999999999 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928603+0000", 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5769e-05 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.040 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0512173459999996, 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013950029999999999, 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8500000000000002e-07 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913567+0000", 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9609000000000001e-05 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913630+0000", 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2220999999999999e-05 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913646+0000", 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6067999999999999e-05 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913738+0000", 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2097999999999998e-05 2024-09-17T16:01:51.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913754+0000", 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5668999999999999e-05 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.914893+0000", 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011396189999999999 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.914933+0000", 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9434000000000002e-05 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0463220470000003, 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020717940000000001, 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6899999999999999e-07 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918457+0000", 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918505+0000", 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7648000000000002e-05 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8609999999999993e-06 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918721+0000", 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020663600000000001 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918732+0000", 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1058e-05 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920477+0000", 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017447840000000001 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920505+0000", 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7815000000000001e-05 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.046030214, 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017398680000000001, 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.052 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918753+0000", 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7325e-05 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918790+0000", 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7764e-05 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918799+0000", 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6459999999999999e-06 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918873+0000", 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4312000000000003e-05 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918883+0000", 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6369999999999994e-06 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920423+0000", 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015398930000000001 2024-09-17T16:01:51.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920465+0000", 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1649000000000001e-05 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:51.057 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.045177797, 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001307234, 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919578+0000", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2200000000000003e-07 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919655+0000", 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7375999999999999e-05 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919705+0000", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9598999999999997e-05 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919716+0000", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1274000000000001e-05 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919782+0000", 2024-09-17T16:01:51.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5438999999999995e-05 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919791+0000", 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.482e-06 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920845+0000", 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010542730000000001 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920884+0000", 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8968999999999999e-05 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0404861439999999, 2024-09-17T16:01:51.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020194549999999999, 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1600000000000002e-07 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924293+0000", 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:01:51.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924390+0000", 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6893999999999998e-05 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924399+0000", 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924465+0000", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309000000000005e-05 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924474+0000", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5289999999999997e-06 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926235+0000", 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001760529 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926288+0000", 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3263000000000001e-05 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0397083949999999, 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016959259999999999, 2024-09-17T16:01:51.067 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:51.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8700000000000004e-07 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3918999999999999e-05 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925131+0000", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.012e-05 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925142+0000", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0481e-05 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925202+0000", 2024-09-17T16:01:51.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9728999999999997e-05 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925211+0000", 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926703+0000", 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014917629999999999 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926742+0000", 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9305999999999999e-05 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=48, reqid=osd.14.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=46'17))", 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925528+0000", 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0392269040000004, 2024-09-17T16:01:51.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001224819, 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925528+0000", 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925528+0000", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925529+0000", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.850000000000001e-07 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925541+0000", 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1664999999999999e-05 2024-09-17T16:01:51.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925575+0000", 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4082000000000003e-05 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925584+0000", 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1840000000000005e-06 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925653+0000", 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9237000000000001e-05 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925662+0000", 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2269999999999999e-06 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926723+0000", 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010609860000000001 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926753+0000", 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9453000000000001e-05 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.077 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:51.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:01:51.080 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:51.077+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:51.179 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:51.177+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:51.187 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:51.187 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:51.187 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:51.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:01:51.280 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:51.279+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:51.343 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:51.343 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:51.344 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:51.344 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:51.344 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:51.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:01:51.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:01:51.382 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:51.380+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:51.482 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:51.480+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:51.502 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:51.502 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:5 3.bs0 3:d4004f6b:::smithi18136759-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.647442615999999, 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010865848000000001, 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.503 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2099999999999997e-07 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852349+0000", 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3249999999999999e-06 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852356+0000", 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6040000000000004e-06 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852361+0000", 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.437e-06 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853722+0000", 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136107 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854082+0000", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036069599999999998 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854755+0000", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067232100000000003 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856366+0000", 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016111039999999999 2024-09-17T16:01:51.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863131+0000", 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067652729999999996 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863211+0000", 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9497000000000001e-05 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.647420391000001, 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:01:51.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.513 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:19 3.1s0 3:8347559a:::smithi18136759-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.647378002, 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010852832999999999, 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 19 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852410+0000", 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852655+0000", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024543600000000003 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852672+0000", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6328e-05 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852682+0000", 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0365e-05 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852763+0000", 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0810999999999996e-05 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852923+0000", 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016075700000000001 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853852+0000", 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092845700000000005 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856526+0000", 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026744960000000002 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863195+0000", 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066689100000000001 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863262+0000", 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6847000000000006e-05 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.647071432000001, 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:01:51.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.647031300999998, 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:46 3.6s0 3:6d55c691:::smithi18136759-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:51.530 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.635541138000001, 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010233671, 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 46 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864246+0000", 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864248+0000", 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7400000000000001e-06 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865054+0000", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000806429 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865069+0000", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4673e-05 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865078+0000", 2024-09-17T16:01:51.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6880000000000005e-06 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865530+0000", 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000452724 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865698+0000", 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016729700000000001 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.866696+0000", 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099818099999999989 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868021+0000", 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013246709999999999 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874396+0000", 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063754190000000002 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.874480+0000", 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3849000000000003e-05 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.634354006999999, 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.536 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:01:51.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.541 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.634080766, 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.542 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:01:51.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.631214984, 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:01:51.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:01:51.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.631163056999998, 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:01:51.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:01:51.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:66 3.6s0 3:60a7d8b5:::smithi18136759-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.619223061, 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010929104, 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880565+0000", 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880568+0000", 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1939999999999998e-06 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880578+0000", 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-06 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880598+0000", 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9964999999999999e-05 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882449+0000", 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018505730000000001 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882543+0000", 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4135999999999995e-05 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890649+0000", 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081062559999999992 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890748+0000", 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9234000000000006e-05 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891493+0000", 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074507600000000001 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.610622290999999, 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:01:51.564 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:01:51.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:01:51.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.610539116999998, 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.570 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:01:51.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.575 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.59609635, 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:01:51.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.596056659999999, 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:51.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:01:51.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:98 3.7s0 3:ec861466:::smithi18136759-33:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.586233559, 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0099957889999999997, 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 98 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913554+0000", 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913555+0000", 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4700000000000001e-07 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913560+0000", 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9969999999999998e-06 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913568+0000", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4740000000000005e-06 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913574+0000", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7309999999999998e-06 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.914995+0000", 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001421381 2024-09-17T16:01:51.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915104+0000", 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010908000000000001 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915686+0000", 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058229899999999999 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916507+0000", 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082023600000000001 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923452+0000", 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069455970000000004 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.923550+0000", 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7246999999999999e-05 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:99 3.7s0 3:ec861466:::smithi18136759-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.586194959, 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011511970999999999, 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.592 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 99 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913593+0000", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913596+0000", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2140000000000001e-06 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913605+0000", 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:01:51.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913612+0000", 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915969+0000", 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023571690000000001 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916049+0000", 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9777000000000005e-05 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924068+0000", 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080191269999999992 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924152+0000", 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4005999999999995e-05 2024-09-17T16:01:51.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.925104+0000", 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095257899999999995 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:122 3.bs0 3:dee30a9d:::smithi18136759-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:51.597 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.556624888999998, 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011179774, 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 122 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:51.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943163+0000", 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2200000000000004e-07 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943166+0000", 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943176+0000", 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.166e-06 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943181+0000", 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.502e-06 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944736+0000", 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015548140000000001 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944814+0000", 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7945e-05 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945207+0000", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039347299999999998 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945618+0000", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041055900000000001 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954241+0000", 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.008622691 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954342+0000", 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010164 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:123 3.bs0 3:dee30a9d:::smithi18136759-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.556596574, 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012212639000000001, 2024-09-17T16:01:51.603 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 123 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943194+0000", 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.565e-06 2024-09-17T16:01:51.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943202+0000", 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0050000000000004e-06 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943207+0000", 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3260000000000003e-06 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945420+0000", 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022128310000000002 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945469+0000", 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8897999999999998e-05 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954714+0000", 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092452019999999992 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954800+0000", 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5541e-05 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.955403+0000", 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060376199999999998 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.608 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.5608144199999998, 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:01:51.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:01:51.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:51.612 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:127: done (5 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:128: done (4 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:129: done (3 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:131: done (2 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:132: done (1 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:133: done (0 left) 2024-09-17T16:01:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:134: snap_create 2024-09-17T16:01:51.615 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:01:51.615 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:51.611+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:51.714 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:51.712+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:51.730 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:51.731 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:51.731 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:51.740 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:01:51.816 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:51.814+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:51.845 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:51.846 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:51.853 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:01:51.916 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:51.915+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.959 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:41 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'19)", 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.032243125000001, 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077219050000000003, 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923885+0000", 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:51.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923947+0000", 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1790000000000003e-05 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923974+0000", 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7368999999999998e-05 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923980+0000", 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924022+0000", 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-09-17T16:01:51.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924037+0000", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4127e-05 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931579+0000", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075422340000000001 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931606+0000", 2024-09-17T16:01:51.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7307000000000001e-05 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:42 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'24, mlcod=25'19)", 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.031720295, 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072286889999999999, 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924408+0000", 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0699999999999997e-07 2024-09-17T16:01:51.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924412+0000", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924432+0000", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9870000000000001e-05 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924436+0000", 2024-09-17T16:01:51.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6e-06 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924471+0000", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4527999999999999e-05 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924487+0000", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6303999999999998e-05 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931609+0000", 2024-09-17T16:01:51.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071217429999999998 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931636+0000", 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6911e-05 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.048904601, 2024-09-17T16:01:51.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.972 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.047434689999999, 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:51.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:01:51.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0410819750000009, 2024-09-17T16:01:51.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:01:51.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:51.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0390993880000003, 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:01:51.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:01:51.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0386735199999997, 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:51.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:51.997 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0386453499999995, 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:51.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:51.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:01:52.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:01:52.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:01:52.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.4s1 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0274821949999993, 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077955969999999996, 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928646+0000", 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:01:52.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928660+0000", 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4535000000000001e-05 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928684+0000", 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3938e-05 2024-09-17T16:01:52.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928692+0000", 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928739+0000", 2024-09-17T16:01:52.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7574999999999998e-05 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928746+0000", 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936413+0000", 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076669930000000004 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936441+0000", 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7413999999999999e-05 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0258444969999996, 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.011 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:01:52.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:01:52.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0255248699999999, 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:01:52.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0254330560000007, 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:01:52.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0366861060000003, 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072413809999999999, 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.025 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919442+0000", 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8500000000000001e-07 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919460+0000", 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7685000000000001e-05 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919508+0000", 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8532999999999998e-05 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919519+0000", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0883000000000001e-05 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920205+0000", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068607399999999995 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920213+0000", 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:01:52.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926416+0000", 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062024070000000001 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926683+0000", 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026676199999999998 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=30, last_complete=46'18, committed=1, applied=1))", 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0360397539999999, 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24942447400000001, 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920091+0000", 2024-09-17T16:01:52.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6620000000000001e-06 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168910+0000", 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24881899499999999 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168924+0000", 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3968000000000001e-05 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168997+0000", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3919999999999997e-05 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.169512+0000", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051455700000000001 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0348279070000004, 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080140109999999997, 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921300+0000", 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921319+0000", 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8519e-05 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2544999999999998e-05 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921346+0000", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9420000000000003e-06 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921389+0000", 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2620000000000002e-05 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921394+0000", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3519999999999997e-06 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929283+0000", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007889007 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929313+0000", 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0460999999999999e-05 2024-09-17T16:01:52.037 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0311228139999997, 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010009911, 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:01:52.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925020+0000", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925042+0000", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2407000000000001e-05 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925050+0000", 2024-09-17T16:01:52.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3089999999999998e-06 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925097+0000", 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7763999999999999e-05 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925103+0000", 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0759999999999999e-06 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098665709999999993 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935014+0000", 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4459000000000002e-05 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.042 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/32 ECSubWrite(tid=186, reqid=osd.11.0:184, at_version=0'0, trim_to=0'0, roll_forward_to=46'22))", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0289524739999996, 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077139770000000003, 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927176+0000", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9400000000000004e-07 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927186+0000", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9259999999999995e-06 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927212+0000", 2024-09-17T16:01:52.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6821999999999998e-05 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927220+0000", 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927269+0000", 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125999999999997e-05 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927278+0000", 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6819999999999992e-06 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934846+0000", 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075681070000000001 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934889+0000", 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2681e-05 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=49, reqid=osd.14.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0289095469999996, 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077062470000000003, 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:52.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5499999999999998e-07 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927227+0000", 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6940000000000001e-06 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927241+0000", 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4260999999999999e-05 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927246+0000", 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6689999999999997e-06 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927292+0000", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927301+0000", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0240000000000003e-06 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934896+0000", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075949219999999996 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934924+0000", 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7620999999999999e-05 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0263526939999998, 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078830700000000007, 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929783+0000", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3150000000000003e-06 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929802+0000", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9959999999999999e-05 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929810+0000", 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3069999999999997e-06 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929880+0000", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0123000000000006e-05 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929890+0000", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937625+0000", 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077345599999999997 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937658+0000", 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2662e-05 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=50, reqid=osd.14.0:50, at_version=46'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.0260641650000002, 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076204879999999999, 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.055 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930064+0000", 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:01:52.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930082+0000", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8311000000000001e-05 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930099+0000", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6895999999999999e-05 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930108+0000", 2024-09-17T16:01:52.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7870000000000007e-06 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930158+0000", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9783000000000001e-05 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930166+0000", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6889999999999993e-06 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937661+0000", 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074941870000000002 2024-09-17T16:01:52.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937684+0000", 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3108999999999998e-05 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.059 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:01:52.061 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:52.058+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:52.162 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:52.160+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:52.205 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.205 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:52.205 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:52.205 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:01:52.262 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:52.260+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:52.317 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:52.318 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:01:52.336 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:01:52.336 INFO:tasks.rados.rados.0.smithi181.stdout:134: done (0 left) 2024-09-17T16:01:52.337 INFO:tasks.rados.rados.0.smithi181.stdout:135: delete oid 28 current snap is 15 2024-09-17T16:01:52.349 INFO:tasks.rados.rados.0.smithi181.stdout:135: done (0 left) 2024-09-17T16:01:52.349 INFO:tasks.rados.rados.0.smithi181.stdout:136: delete oid 8 current snap is 15 2024-09-17T16:01:52.359 INFO:tasks.rados.rados.0.smithi181.stdout:136: done (0 left) 2024-09-17T16:01:52.359 INFO:tasks.rados.rados.0.smithi181.stdout:137: delete oid 38 current snap is 15 2024-09-17T16:01:52.363 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:52.361+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:52.366 INFO:tasks.rados.rados.0.smithi181.stdout:137: done (0 left) 2024-09-17T16:01:52.367 INFO:tasks.rados.rados.0.smithi181.stdout:138: snap_create 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.440 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.552970903, 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038886110000000002, 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884707+0000", 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2036999999999999e-05 2024-09-17T16:01:52.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884745+0000", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7843999999999999e-05 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884756+0000", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1240000000000001e-05 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884826+0000", 2024-09-17T16:01:52.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9873e-05 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884836+0000", 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0053e-05 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888486+0000", 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036497270000000002 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888573+0000", 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7068000000000001e-05 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.551361108, 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.445 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:01:52.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.550549865000001, 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044954000000000001, 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4892000000000002e-05 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887167+0000", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5617999999999997e-05 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887176+0000", 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6379999999999993e-06 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887237+0000", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1760000000000005e-05 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887246+0000", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891572+0000", 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004325626 2024-09-17T16:01:52.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891601+0000", 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9105e-05 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.5182877109999993, 2024-09-17T16:01:52.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027013699999999998, 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919368+0000", 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.58e-07 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919591+0000", 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022241000000000001 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919654+0000", 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3744999999999994e-05 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919663+0000", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7909999999999993e-06 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919742+0000", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8751999999999994e-05 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919753+0000", 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.111e-05 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922038+0000", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002285418 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922069+0000", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0685999999999997e-05 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 45/32 ECSubWrite(tid=27, reqid=client.4560.0:266, at_version=45'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:52.459 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.5152446229999992, 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002821306, 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922411+0000", 2024-09-17T16:01:52.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922412+0000", 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0800000000000004e-07 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922427+0000", 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5136999999999999e-05 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922462+0000", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5172999999999997e-05 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922471+0000", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922539+0000", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8458000000000001e-05 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922551+0000", 2024-09-17T16:01:52.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2106999999999999e-05 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925193+0000", 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002641539 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925232+0000", 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9298000000000002e-05 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 45/32 ECSubWrite(tid=66, reqid=client.4560.0:275, at_version=0'0, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.5043525469999999, 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034407219999999998, 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933303+0000", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3000000000000001e-07 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933311+0000", 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0229999999999992e-06 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933334+0000", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2843000000000001e-05 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933340+0000", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.524e-06 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933385+0000", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5037000000000003e-05 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.933392+0000", 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936713+0000", 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0033208700000000001 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936744+0000", 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0935000000000001e-05 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.51634694, 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024618610000000001, 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921309+0000", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7199999999999999e-07 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921323+0000", 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3711e-05 2024-09-17T16:01:52.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921350+0000", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7110999999999998e-05 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921355+0000", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5060000000000003e-06 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921406+0000", 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0337000000000003e-05 2024-09-17T16:01:52.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921415+0000", 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6350000000000001e-06 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923739+0000", 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023234620000000001 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923770+0000", 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1526999999999998e-05 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 46/32 ECSubWrite(tid=69, reqid=osd.13.0:68, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5143393119999997, 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030339099999999999, 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.472 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923316+0000", 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923317+0000", 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923325+0000", 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4279999999999996e-06 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923347+0000", 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1980999999999998e-05 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923353+0000", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3900000000000001e-06 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923391+0000", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8013999999999999e-05 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.923396+0000", 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1529999999999996e-06 2024-09-17T16:01:52.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926319+0000", 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002923602 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926350+0000", 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0716000000000002e-05 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.513556232, 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054480279999999997, 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:52.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924122+0000", 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2844999999999999e-05 2024-09-17T16:01:52.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924159+0000", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6534e-05 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924166+0000", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.742e-06 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924210+0000", 2024-09-17T16:01:52.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4641999999999997e-05 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924220+0000", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7580000000000004e-06 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929512+0000", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005292265 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929547+0000", 2024-09-17T16:01:52.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5046999999999998e-05 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=51, reqid=osd.12.0:51, at_version=46'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5134490940000003, 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053997719999999997, 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924207+0000", 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.482 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924223+0000", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6835000000000001e-05 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924252+0000", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8285000000000001e-05 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924259+0000", 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:01:52.483 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924320+0000", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1709000000000002e-05 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924329+0000", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8799999999999997e-06 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929582+0000", 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052531169999999999 2024-09-17T16:01:52.484 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929606+0000", 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3913e-05 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5126268539999996, 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.485 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.486 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.487 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:01:52.488 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.489 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 46/32 ECSubWrite(tid=70, reqid=osd.13.0:67, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5122994680000001, 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042172039999999996, 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.490 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925357+0000", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925366+0000", 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-09-17T16:01:52.491 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925396+0000", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9964e-05 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925404+0000", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6429999999999995e-06 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925470+0000", 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6817999999999996e-05 2024-09-17T16:01:52.492 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925480+0000", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929543+0000", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040623580000000003 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929573+0000", 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0682000000000002e-05 2024-09-17T16:01:52.493 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5102678919999999, 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:52.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.495 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5078672500000003, 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052541769999999996, 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929789+0000", 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929800+0000", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1341e-05 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929829+0000", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8642000000000001e-05 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929837+0000", 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929901+0000", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4067999999999997e-05 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929907+0000", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.609e-06 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934990+0000", 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050827579999999997 2024-09-17T16:01:52.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935042+0000", 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2676999999999997e-05 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=39, reqid=osd.0.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.5051070959999997, 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050006720000000003, 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.502 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932549+0000", 2024-09-17T16:01:52.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932561+0000", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1674e-05 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932581+0000", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0024999999999999e-05 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932590+0000", 2024-09-17T16:01:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7780000000000002e-06 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932642+0000", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1753999999999999e-05 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932651+0000", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2310000000000002e-06 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937518+0000", 2024-09-17T16:01:52.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048669969999999996 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937549+0000", 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0651999999999997e-05 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=54, reqid=osd.12.0:53, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.4996377609999998, 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028466110000000002, 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.506 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938018+0000", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2600000000000002e-07 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938026+0000", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9209999999999992e-06 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938055+0000", 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8252999999999999e-05 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938061+0000", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0249999999999999e-06 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938102+0000", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1875000000000001e-05 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.938109+0000", 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4200000000000004e-06 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940826+0000", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002716754 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940864+0000", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8736999999999999e-05 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 52/32 ECSubWrite(tid=48, reqid=osd.2.0:48, at_version=52'35, trim_to=0'0, roll_forward_to=51'31))", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.348743+0000", 2024-09-17T16:01:52.510 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.0889120320000001, 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030576319999999998, 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348743+0000", 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348743+0000", 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.511 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348744+0000", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9400000000000004e-07 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348770+0000", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6023000000000001e-05 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348811+0000", 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0902999999999998e-05 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348819+0000", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5159999999999997e-06 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348901+0000", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2158999999999997e-05 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.348915+0000", 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3584000000000001e-05 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351760+0000", 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028457700000000001 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351801+0000", 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0782999999999999e-05 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.514 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.0863990139999999, 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043350350000000001, 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:01:52.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351257+0000", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0100000000000001e-07 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351295+0000", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8333e-05 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351337+0000", 2024-09-17T16:01:52.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1491e-05 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351345+0000", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5189999999999992e-06 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351424+0000", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351434+0000", 2024-09-17T16:01:52.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355546+0000", 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041123100000000001 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355591+0000", 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5212e-05 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 52/32 ECSubWrite(tid=49, reqid=osd.2.0:48, at_version=0'0, trim_to=0'0, roll_forward_to=52'35))", 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.354846+0000", 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.0828094109999999, 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0030355550000000001, 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354846+0000", 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354846+0000", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354847+0000", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354868+0000", 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1209999999999999e-05 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354920+0000", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1796999999999999e-05 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.354931+0000", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0397999999999999e-05 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355012+0000", 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1589999999999996e-05 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355022+0000", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8050000000000001e-06 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.357831+0000", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028093979999999998 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.357882+0000", 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0219000000000001e-05 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.522 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=85, reqid=osd.13.0:78, at_version=0'0, trim_to=0'0, roll_forward_to=52'18))", 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.067314213, 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037161669999999998, 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.17e-07 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370349+0000", 2024-09-17T16:01:52.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4499999999999998e-06 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370413+0000", 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4089000000000003e-05 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370419+0000", 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2740000000000004e-06 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370550+0000", 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131163 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370561+0000", 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374029+0000", 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034684590000000001 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374057+0000", 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8064000000000001e-05 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.527 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.528 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:01:52.529 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:52.526+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:52.629 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:52.627+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:52.632 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.633 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:52.633 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:52.633 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.641 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:01:52.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:52.730 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:52.728+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:52.744 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:52.744 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.752 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:01:52.802 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:52.802 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:52.831 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:52.829+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:52.856 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:52.856 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:52.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:52.856 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:52.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.998667186999999, 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:52.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.997766313, 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.993676485999998, 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:01:52.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.992338039, 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055595449999999999, 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.870 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861313+0000", 2024-09-17T16:01:52.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9800000000000002e-07 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861333+0000", 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0276999999999999e-05 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861372+0000", 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9365e-05 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861382+0000", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3349999999999996e-06 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861456+0000", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3972000000000003e-05 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861465+0000", 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7480000000000005e-06 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866841+0000", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053751720000000001 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866871+0000", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0678e-05 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:52.874 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.991372939000001, 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045470969999999999, 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:01:52.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862278+0000", 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5199999999999995e-07 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862373+0000", 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5013999999999995e-05 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.876 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862431+0000", 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862440+0000", 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6339999999999996e-06 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862513+0000", 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3028999999999995e-05 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862522+0000", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866779+0000", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042571140000000002 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866824+0000", 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4511000000000001e-05 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.878 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.991269525, 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:01:52.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.989444147, 2024-09-17T16:01:52.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:52.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:01:52.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:01:52.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.989296071999998, 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.888 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:01:52.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.892 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.988646188000001, 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:01:52.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:01:52.896 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.986068825, 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.985577653, 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=6, reqid=client.4560.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.984840802000001, 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048136530000000002, 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.905 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868810+0000", 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9099999999999991e-07 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868951+0000", 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014129799999999999 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868987+0000", 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5556000000000003e-05 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868996+0000", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6549999999999999e-06 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869062+0000", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5714000000000001e-05 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869072+0000", 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0273e-05 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873569+0000", 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044970189999999997 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873623+0000", 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3146999999999998e-05 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.909 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9285787250000004, 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043002459999999998, 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925072+0000", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9500000000000002e-07 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925120+0000", 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8337999999999997e-05 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925170+0000", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9965e-05 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925178+0000", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2220000000000002e-06 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925233+0000", 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4378e-05 2024-09-17T16:01:52.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925239+0000", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2090000000000002e-06 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929339+0000", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041006410000000004 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929371+0000", 2024-09-17T16:01:52.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1798000000000003e-05 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9285265349999996, 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042998380000000003, 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925123+0000", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925124+0000", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8299999999999998e-07 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925141+0000", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7156e-05 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925203+0000", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1990999999999997e-05 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925208+0000", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7230000000000001e-06 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925267+0000", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8118000000000002e-05 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925275+0000", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9299999999999992e-06 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929405+0000", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041293140000000002 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929423+0000", 2024-09-17T16:01:52.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8223000000000002e-05 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 46/32 ECSubWrite(tid=40, reqid=osd.0.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9181742330000002, 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005398712, 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935476+0000", 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6000000000000004e-07 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935486+0000", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935508+0000", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2333000000000001e-05 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935516+0000", 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8779999999999998e-06 2024-09-17T16:01:52.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935579+0000", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3529999999999997e-05 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935588+0000", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9560000000000003e-06 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940834+0000", 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052456009999999999 2024-09-17T16:01:52.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940874+0000", 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0224e-05 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 52/32 ECSubWrite(tid=82, reqid=client.4560.0:335, at_version=52'11, trim_to=0'0, roll_forward_to=52'8))", 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.489901041, 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048268470000000004, 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.922 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8800000000000002e-07 2024-09-17T16:01:52.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363764+0000", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4596000000000001e-05 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363789+0000", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4715e-05 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363795+0000", 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0619999999999997e-06 2024-09-17T16:01:52.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363861+0000", 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6214e-05 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363873+0000", 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2153e-05 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.368548+0000", 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046749959999999998 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.368576+0000", 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7423000000000002e-05 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 52/32 ECSubWrite(tid=84, reqid=client.4560.0:335, at_version=0'0, trim_to=0'0, roll_forward_to=52'11))", 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:01:52.926 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.4845121880000001, 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048528909999999998, 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:01:52.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1200000000000003e-07 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369146+0000", 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7160000000000003e-06 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371657+0000", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025113610000000001 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371664+0000", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6869999999999997e-06 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371713+0000", 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8724999999999999e-05 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371719+0000", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-06 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373960+0000", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022413239999999998 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373990+0000", 2024-09-17T16:01:52.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0496000000000001e-05 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 52/32 ECSubWrite(tid=89, reqid=client.4560.0:340, at_version=52'11, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 2.4806562429999999, 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15958299200000001, 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372994+0000", 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373011+0000", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6280999999999999e-05 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531935+0000", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15892432300000001 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531949+0000", 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532026+0000", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6526000000000006e-05 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532033+0000", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532538+0000", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050494099999999996 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532576+0000", 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8059999999999998e-05 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 54/32 ECSubWrite(tid=53, reqid=client.4560.0:344, at_version=54'24, trim_to=0'0, roll_forward_to=52'22))", 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.51098311900000004, 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045033649999999996, 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:01:52.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342688+0000", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1129e-05 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342910+0000", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022252599999999999 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342920+0000", 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4749999999999999e-06 2024-09-17T16:01:52.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342980+0000", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0260999999999998e-05 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342990+0000", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4709999999999996e-06 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.347131+0000", 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041411149999999999 2024-09-17T16:01:52.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.347170+0000", 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9205000000000002e-05 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 54/32 ECSubWrite(tid=54, reqid=client.4560.0:344, at_version=0'0, trim_to=0'0, roll_forward_to=54'24))", 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.50544450600000002, 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046180140000000001, 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:52.939 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:52.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348206+0000", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7700000000000004e-07 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348215+0000", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.268e-06 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348242+0000", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7356e-05 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:52.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348252+0000", 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0018e-05 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348318+0000", 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5888000000000003e-05 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348330+0000", 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1232e-05 2024-09-17T16:01:52.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.352784+0000", 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044541750000000003 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.352823+0000", 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9499999999999998e-05 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:52.943 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:52.945 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:52.943+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:52.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:01:53.027 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:53.028 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:53.046 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:53.044+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:53.049 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:53.049 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:53.071 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:53.071 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:53.091 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:53.091 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:53.091 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:53.091 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:53.092 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:53.092 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:53.100 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:01:53.147 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:53.145+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:53.203 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:53.203 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:53.203 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:53.203 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:53.203 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:53.212 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:01:53.248 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:53.246+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:53.316 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:8 3.0s0 3:03e99693:::smithi18136759-3:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.461191398, 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.011147927, 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.317 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 8 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852743+0000", 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852744+0000", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.159e-06 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852751+0000", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9120000000000001e-06 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852762+0000", 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1192000000000001e-05 2024-09-17T16:01:53.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4889999999999998e-06 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854588+0000", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018183909999999999 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854806+0000", 2024-09-17T16:01:53.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000217347 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855286+0000", 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048023199999999998 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.857505+0000", 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022190869999999998 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863759+0000", 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062543 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863891+0000", 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013181799999999999 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.461151059999999, 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:01:53.322 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:01:53.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.327 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.461122546999999, 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:01:53.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:01:53.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.460983502000001, 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.333 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:53.338 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.460955827999999, 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:01:53.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs2 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.455148640000001, 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012875312, 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858786+0000", 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6400000000000001e-07 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858933+0000", 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014714799999999999 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858998+0000", 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5139000000000001e-05 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859014+0000", 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5814000000000001e-05 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859094+0000", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9672000000000006e-05 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859104+0000", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0596e-05 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871628+0000", 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012524086 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871661+0000", 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2693e-05 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:39 3.9s0 3:905994b3:::smithi18136759-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.454975978, 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013438131000000001, 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.348 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 39 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858958+0000", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858959+0000", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3399999999999998e-07 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858963+0000", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.427e-06 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858970+0000", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5270000000000001e-06 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858977+0000", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9399999999999996e-06 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861610+0000", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0026333149999999998 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:53.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861663+0000", 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2731e-05 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868368+0000", 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067046069999999996 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868428+0000", 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0503e-05 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.872397+0000", 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039682470000000003 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=2, last_complete=34'3, committed=1, applied=1))", 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.450379584, 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014226245, 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.353 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:01:53.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863557+0000", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1169999999999998e-06 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863638+0000", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.03e-05 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863644+0000", 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4150000000000004e-06 2024-09-17T16:01:53.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877622+0000", 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013978305 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877781+0000", 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015883299999999999 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.450375913999999, 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014337127, 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863559+0000", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5800000000000001e-07 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863560+0000", 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863674+0000", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011379000000000001 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863682+0000", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877875+0000", 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014193012 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877896+0000", 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0993e-05 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s4 35/32 ECSubWrite(tid=119, reqid=client.4560.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.449237093000001, 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015598302, 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864698+0000", 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.44e-07 2024-09-17T16:01:53.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864708+0000", 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.73e-06 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864735+0000", 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7804e-05 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864741+0000", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7969999999999997e-06 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877824+0000", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01308231 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877833+0000", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4809999999999995e-06 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880272+0000", 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024384799999999998 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880296+0000", 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4056000000000001e-05 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.396519250000001, 2024-09-17T16:01:53.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:01:53.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:01:53.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.396494046000001, 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.370 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.393128052000002, 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.375 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:01:53.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:01:53.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.366153402999998, 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:01:53.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.366135532000001, 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.388 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 19.362841764999999, 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:185 3.9s0 3:99c5a4ac:::smithi18136759-42:head [copy-from ver 7] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.441003557, 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019207894999999999, 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.392 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 185 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:01:53.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872932+0000", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.635e-06 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872938+0000", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1680000000000003e-06 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872948+0000", 2024-09-17T16:01:53.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0477e-05 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872955+0000", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7750000000000004e-06 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873015+0000", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873024+0000", 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9439999999999994e-06 2024-09-17T16:01:53.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874582+0000", 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00155798 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874682+0000", 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100215 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:53.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887716+0000", 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013033741999999999 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889595+0000", 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018794879999999999 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892089+0000", 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024933450000000001 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892139+0000", 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0172e-05 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:186 3.9s0 3:99c5a4ac:::smithi18136759-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.440966921999999, 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016243253999999999, 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.398 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 186 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872968+0000", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2500000000000001e-07 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872969+0000", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.525e-06 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872975+0000", 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.186e-06 2024-09-17T16:01:53.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872978+0000", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.202e-06 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873075+0000", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7590999999999994e-05 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873079+0000", 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.54e-06 2024-09-17T16:01:53.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874867+0000", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017885100000000001 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for blocked object", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874930+0000", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2175999999999996e-05 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889013+0000", 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014083304 2024-09-17T16:01:53.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889112+0000", 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8972e-05 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889211+0000", 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8923000000000001e-05 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:01:53.403 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.362726009999999, 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:01:53.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.3828726299999996, 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:53.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:53.410 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:53.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:01:53.414 INFO:tasks.rados.rados.0.smithi181.stdout:138: done (0 left) 2024-09-17T16:01:53.414 INFO:tasks.rados.rados.0.smithi181.stdout:139: read oid 49 snap -1 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:139: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:140: copy_from oid 31 from oid 19 current snap is 16 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 29 current snap is 16 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:141: seq_num 77 ranges {475136=524288,999424=524288,1523712=245760} 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:141: writing smithi18136759-29 from 475136 to 999424 tid 1 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:141: writing smithi18136759-29 from 999424 to 1523712 tid 2 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:141: writing smithi18136759-29 from 1523712 to 1769472 tid 3 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:142: copy_from oid 24 from oid 44 current snap is 16 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:139: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 694189 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 733749 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 435991 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:140: finishing copy_from racing read to smithi18136759-31 2024-09-17T16:01:53.415 INFO:tasks.rados.rados.0.smithi181.stdout:139: done (3 left) 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:143: read oid 2 snap 7 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:143: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:144: read oid 9 snap 14 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:144: expect (ObjNum 56 snap 6 seq_num 56) 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:145: setattr oid 47 current snap is 16 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:140: finishing copy_from to smithi18136759-31 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing write tid 1 to smithi18136759-29 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing write tid 2 to smithi18136759-29 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing write tid 3 to smithi18136759-29 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:140: done (5 left) 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:146: copy_from oid 48 from oid 8 current snap is 16 2024-09-17T16:01:53.416 INFO:tasks.rados.rados.0.smithi181.stdout:147: snap_remove snap 15 2024-09-17T16:01:53.417 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:53.410+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:53.512 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:53.510+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:53.527 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:53.527 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:53.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:01:53.613 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:53.612+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:53.638 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:53.638 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:53.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:01:53.714 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:53.712+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:53.789 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:53.789 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:53.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:01:53.815 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:53.813+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:53.900 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:01:53.900 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:01:53.901 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:01:53.901 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:01:53.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-17T16:01:53.917 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:53.918 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:53.940 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:53.940 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:53.953 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:53.952+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:53.961 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:53.961 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:53.982 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:53.982 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:54.005 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:54.005 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:54.013 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.013 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.017 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:54.015+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:54.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-17T16:01:54.027 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:54.027 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:54.049 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:54.049 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:54.069 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:54.069 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:54.091 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:54.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:54.091 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:54.090+0000 7febc531d640 0 -- 172.21.15.6:0/2724627554 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7febc0221230 msgr2=0x7febc0223620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:54.118 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:54.116+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:54.164 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.164 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:01:54.218 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:54.216+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:54.276 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.276 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:01:54.316 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:54.316 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:54.319 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:54.317+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:54.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:54.338 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:54.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:01:54.367 INFO:tasks.rados.rados.0.smithi181.stdout:147: done (6 left) 2024-09-17T16:01:54.367 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 30 current snap is 16 2024-09-17T16:01:54.367 INFO:tasks.rados.rados.0.smithi181.stdout:148: seq_num 78 ranges {1720320=557056,2277376=671744,2949120=98304} 2024-09-17T16:01:54.379 INFO:tasks.rados.rados.0.smithi181.stdout:148: writing smithi18136759-30 from 1720320 to 2277376 tid 1 2024-09-17T16:01:54.383 INFO:tasks.rados.rados.0.smithi181.stdout:148: writing smithi18136759-30 from 2277376 to 2949120 tid 2 2024-09-17T16:01:54.384 INFO:tasks.rados.rados.0.smithi181.stdout:148: writing smithi18136759-30 from 2949120 to 3047424 tid 3 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:149: snap_create 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:142: finishing copy_from racing read to smithi18136759-24 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing write tid 4 to smithi18136759-29 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:143: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 25814 2024-09-17T16:01:54.385 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:54.386 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 32740 2024-09-17T16:01:54.386 INFO:tasks.rados.rados.0.smithi181.stdout:141: finishing write tid 5 to smithi18136759-29 2024-09-17T16:01:54.386 INFO:tasks.rados.rados.0.smithi181.stdout:141: oid 29 updating version 0 to 47 2024-09-17T16:01:54.386 INFO:tasks.rados.rados.0.smithi181.stdout:141: oid 29 version 47 is already newer than 46 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:141: oid 29 version 47 is already newer than 45 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:141: oid 29 updating version 47 to 48 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 29 v 48 (ObjNum 77 snap 16 seq_num 77) dirty exists 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:141: left oid 29 (ObjNum 77 snap 16 seq_num 77) 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:146: finishing copy_from to smithi18136759-48 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:146: got expected ENOENT (src dne) 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:146: finishing copy_from racing read to smithi18136759-48 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:144: expect (ObjNum 56 snap 6 seq_num 56) 2024-09-17T16:01:54.387 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:54.395 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.395 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.401 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 705954 2024-09-17T16:01:54.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:01:54.405 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:54.416 INFO:tasks.rados.rados.0.smithi181.stdout:142: finishing copy_from to smithi18136759-24 2024-09-17T16:01:54.416 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 28 (ObjNum 59 snap 7 seq_num 59) dirty exists 2024-09-17T16:01:54.416 INFO:tasks.rados.rados.0.smithi181.stdout:148: finishing write tid 1 to smithi18136759-30 2024-09-17T16:01:54.416 INFO:tasks.rados.rados.0.smithi181.stdout:148: finishing write tid 2 to smithi18136759-30 2024-09-17T16:01:54.416 INFO:tasks.rados.rados.0.smithi181.stdout:148: finishing write tid 3 to smithi18136759-30 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: finishing write tid 4 to smithi18136759-30 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: finishing write tid 5 to smithi18136759-30 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: oid 30 updating version 0 to 14 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: oid 30 version 14 is already newer than 13 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: oid 30 updating version 14 to 16 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: oid 30 version 16 is already newer than 15 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 16 (ObjNum 78 snap 16 seq_num 78) dirty exists 2024-09-17T16:01:54.417 INFO:tasks.rados.rados.0.smithi181.stdout:148: left oid 30 (ObjNum 78 snap 16 seq_num 78) 2024-09-17T16:01:54.436 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:54.434+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:54.506 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.506 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:01:54.522 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:54.519+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:54.621 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:54.620+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:54.656 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.656 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:01:54.723 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:54.721+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:54.769 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.769 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:01:54.823 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:01:54.821+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:01:54.880 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.881 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:54.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:01:54.924 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:54.922+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:54.997 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:54.997 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:01:55.025 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:55.023+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:55.113 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.113 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:01:55.125 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:55.123+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:55.226 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:55.224+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:55.228 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.228 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:01:55.327 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:55.325+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:55.344 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.344 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:01:55.362 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-17T16:01:55.362 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:01:55.363 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:01:55.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T16:01:55.369 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:141: done (7 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:142: done (6 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:143: done (5 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:144: done (4 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:145: done (3 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:146: done (2 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:148: done (1 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:149: done (0 left) 2024-09-17T16:01:55.373 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 33 current snap is 17 2024-09-17T16:01:55.374 INFO:tasks.rados.rados.0.smithi181.stdout:150: seq_num 79 ranges {540672=573440,1114112=425984,1540096=344064} 2024-09-17T16:01:55.379 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18136759-33 from 540672 to 1114112 tid 1 2024-09-17T16:01:55.381 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18136759-33 from 1114112 to 1540096 tid 2 2024-09-17T16:01:55.383 INFO:tasks.rados.rados.0.smithi181.stdout:150: writing smithi18136759-33 from 1540096 to 1884160 tid 3 2024-09-17T16:01:55.383 INFO:tasks.rados.rados.0.smithi181.stdout:151: setattr oid 21 current snap is 17 2024-09-17T16:01:55.385 INFO:tasks.rados.rados.0.smithi181.stdout:152: setattr oid 27 current snap is 17 2024-09-17T16:01:55.387 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 1 to smithi18136759-33 2024-09-17T16:01:55.387 INFO:tasks.rados.rados.0.smithi181.stdout:153: snap_remove snap 16 2024-09-17T16:01:55.428 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:55.426+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:55.466 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.466 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:01:55.528 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:55.527+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:55.588 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.588 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:01:55.629 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:55.627+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:55.694 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:55.694 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:01:55.694 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 57, 2024-09-17T16:01:55.694 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "4345ca04-2628-4bef-a537-6837268b96c9", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-17T16:01:02.569356+0000", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-17T16:01:55.363347+0000", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-09-17T16:01:15.727345+0000", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-09-17T16:01:06.794253+0000", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5806080, 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "nodeep-scrub", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "noscrub", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-09-17T16:01:55.695 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 7, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 3, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 16, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "jewel", 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.696 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:14.963100+0000", 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 2, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 1, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:01:55.697 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 1, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 1, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 1, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 1, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 1, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:01:55.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "25", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:01:55.699 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:01:55.700 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:01:55.701 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 15.380000114440918, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 15.380000114440918, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.12999999523162842, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2, 2024-09-17T16:01:55.702 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "rbd", 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:17.236630+0000", 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-09-17T16:01:55.703 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 2, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 1, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 8, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 8, 2024-09-17T16:01:55.704 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 8, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 8, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 8, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "28", 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:01:55.705 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 28, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:01:55.706 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:01:55.707 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-09-17T16:01:55.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 6.0399999618530273, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 6.0399999618530273, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.56000000238418579, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 3.380000114440918, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 3.380000114440918, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.710 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-09-17T16:01:55.710 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_0", 2024-09-17T16:01:55.710 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:29.829066+0000", 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 3, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 7, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 5, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 1, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:01:55.711 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 26, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 17, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 26, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 26, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 26, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:01:55.712 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "57", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "53", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 18, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 57, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:01:55.713 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "shecprofile", 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:01:55.714 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 16384, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:01:55.715 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "rados": {} 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "282c4269-7e53-43fc-af4b-3b133d82837c", 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.716 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 32, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-09-17T16:01:55.718 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.719 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.720 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/865094279", 2024-09-17T16:01:55.721 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6811/865094279", 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6815/865094279", 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6813/865094279", 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "66e20d10-2bb0-4130-93a5-45bcdd2de605", 2024-09-17T16:01:55.722 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 21, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.723 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6801", 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6803", 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.724 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6805", 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6807", 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.725 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6813", 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6815", 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.726 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6809", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6811", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6803/1460756480", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6807/1460756480", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1460756480", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6811/1460756480", 2024-09-17T16:01:55.727 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "c6ec9e20-133a-471b-b63c-6453a7bde86c", 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.728 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 32, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6816", 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6817", 2024-09-17T16:01:55.729 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6818", 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6819", 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.730 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6822", 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6823", 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.731 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6820", 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6821", 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:01:55.732 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6817/1905413450", 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6819/1905413450", 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6823/1905413450", 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6821/1905413450", 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.733 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3491a731-ca96-4981-be73-e005e00c315f", 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.734 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6810", 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6811", 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.735 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6812", 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6813", 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.736 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6816", 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6817", 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.737 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6814", 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6815", 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.738 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6811/2918557179", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6813/2918557179", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6817/2918557179", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6815/2918557179", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3bdfaa34-bc10-4cec-b5c2-86e05d444ffd", 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.739 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6800", 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.740 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.741 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.742 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.743 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/2257685015", 2024-09-17T16:01:55.744 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6803/2257685015", 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6807/2257685015", 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6805/2257685015", 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2", 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.745 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.746 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6824", 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6825", 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.747 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6826", 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6827", 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6830", 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6831", 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.749 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6828", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6829", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6825/2399951648", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6827/2399951648", 2024-09-17T16:01:55.750 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6831/2399951648", 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6829/2399951648", 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "ddd3edea-260a-4ec9-8533-841ed933fdb8", 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.751 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 25, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6824", 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.752 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6825", 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6826", 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.753 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6827", 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6830", 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.754 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6831", 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6828", 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6829", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6825/1225429423", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6827/1225429423", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6831/1225429423", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6829/1225429423", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.756 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7afc1a18-98c2-4a58-a09d-1b4f474d4dc4", 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.757 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6818", 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6819", 2024-09-17T16:01:55.758 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6820", 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.759 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6821", 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6824", 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.760 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6825", 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6822", 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.761 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6823", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6819/3415295348", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6821/3415295348", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6825/3415295348", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6823/3415295348", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.762 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 8, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3e45f80f-856b-4d3f-b002-255f16a14f6b", 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.763 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6824", 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6825", 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.764 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6826", 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6827", 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.765 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6830", 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6831", 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6828", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6829", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6825/2216439459", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6827/2216439459", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6831/2216439459", 2024-09-17T16:01:55.767 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6829/2216439459", 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 9, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b415c423-24de-4d86-bec4-ea330364ff96", 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.768 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 25, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6816", 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.769 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6817", 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6818", 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.770 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6819", 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6822", 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.771 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6823", 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6820", 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.772 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6821", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6817/1727176716", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6819/1727176716", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6823/1727176716", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6821/1727176716", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.773 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 10, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "e471e1cb-9c0b-4c99-8622-59eb7956770d", 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:01:55.774 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6808", 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6809", 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.775 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6810", 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6811", 2024-09-17T16:01:55.776 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6814", 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.777 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6815", 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6812", 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.778 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6813", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6809/3604183526", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6811/3604183526", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3604183526", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6813/3604183526", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 11, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "c2bf4255-7ce1-4673-bedf-fd055d841fac", 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.780 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6826", 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6827", 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.781 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6828", 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6829", 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.782 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6832", 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6833", 2024-09-17T16:01:55.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6830", 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.784 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6831", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6827/3420013159", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6829/3420013159", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6833/3420013159", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6831/3420013159", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 12, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "4c16f3ee-f427-41d5-b0ba-9e676ddbe032", 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 23, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 32, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.786 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6817", 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6818", 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6819", 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6822", 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6823", 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.789 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6820", 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6821", 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:01:55.790 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6817/2310487797", 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6819/2310487797", 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6823/2310487797", 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6821/2310487797", 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.791 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 13, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "0a4386aa-1a59-44c8-8d1e-7d43bf10de2e", 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6800", 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6802", 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.793 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6804", 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6806", 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6812", 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6814", 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.795 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6808", 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6810", 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:01:55.796 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6802/1210437211", 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6806/1210437211", 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6814/1210437211", 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6810/1210437211", 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.797 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 14, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "4c54e35c-e226-4d8c-bad0-51dd1cd95fe0", 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6800", 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6801", 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.799 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6802", 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6803", 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6806", 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6807", 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.801 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6804", 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6805", 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.802 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6801/4170280864", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6803/4170280864", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6807/4170280864", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6805/4170280864", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 15, 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "0281e750-f74a-4760-bbef-fbfa4127d06b", 2024-09-17T16:01:55.803 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.804 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6802", 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6803", 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.805 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6804", 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6805", 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6808", 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6809", 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:01:55.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6806", 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6807", 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6803/2815528414", 2024-09-17T16:01:55.808 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6805/2815528414", 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6809/2815528414", 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6807/2815528414", 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.809 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.014883+0000", 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.810 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.677746+0000", 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.838957+0000", 2024-09-17T16:01:55.811 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.321698+0000", 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.812 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.045654+0000", 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.813 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.762032+0000", 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.517939+0000", 2024-09-17T16:01:55.814 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.664321+0000", 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.815 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 8, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.417977+0000", 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 9, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.816 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.810458+0000", 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 10, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.817 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.124128+0000", 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 11, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.554475+0000", 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.818 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 12, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.868839+0000", 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 13, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.819 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.912548+0000", 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 14, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.882601+0000", 2024-09-17T16:01:55.820 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 15, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.090747+0000", 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:01:55.821 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [], 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [], 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [], 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": {}, 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-09-17T16:01:55.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "shecprofile": { 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "c": "2", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-device-class": "", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-num-failure-domains": "0", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-osds-per-failure-domain": "0", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-root": "default", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "4", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "3", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "shecprofile", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "shec", 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "multiple" 2024-09-17T16:01:55.823 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [ 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "snaps": [ 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 7, 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 1 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 11, 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 1 2024-09-17T16:01:55.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 17, 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 1 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-09-17T16:01:55.825 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-09-17T16:01:55.826 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:01:55.826 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:01:55.826 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-09-17T16:01:55.826 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:01:55.826 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:01:55.828 INFO:tasks.thrashosds.thrasher:j is {'epoch': 57, 'fsid': '4345ca04-2628-4bef-a537-6837268b96c9', 'created': '2024-09-17T16:01:02.569356+0000', 'modified': '2024-09-17T16:01:55.363347+0000', 'last_up_change': '2024-09-17T16:01:15.727345+0000', 'last_in_change': '2024-09-17T16:01:06.794253+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:01:14.963100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-17T16:01:17.236630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.039999961853027, 'score_stable': 6.039999961853027, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 3.380000114440918, 'raw_score_stable': 3.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-17T16:01:29.829066+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 17, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '53', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 18, 'snap_epoch': 57, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '282c4269-7e53-43fc-af4b-3b133d82837c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6809', 'nonce': 865094279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6811', 'nonce': 865094279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6815', 'nonce': 865094279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6813', 'nonce': 865094279}]}, 'public_addr': '172.21.15.6:6809/865094279', 'cluster_addr': '172.21.15.6:6811/865094279', 'heartbeat_back_addr': '172.21.15.6:6815/865094279', 'heartbeat_front_addr': '172.21.15.6:6813/865094279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '66e20d10-2bb0-4130-93a5-45bcdd2de605', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1460756480}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 1460756480}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1460756480}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 1460756480}]}, 'public_addr': '172.21.15.50:6803/1460756480', 'cluster_addr': '172.21.15.50:6807/1460756480', 'heartbeat_back_addr': '172.21.15.50:6815/1460756480', 'heartbeat_front_addr': '172.21.15.50:6811/1460756480', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c6ec9e20-133a-471b-b63c-6453a7bde86c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6816', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6817', 'nonce': 1905413450}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6818', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6819', 'nonce': 1905413450}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6822', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6823', 'nonce': 1905413450}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6820', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6821', 'nonce': 1905413450}]}, 'public_addr': '172.21.15.169:6817/1905413450', 'cluster_addr': '172.21.15.169:6819/1905413450', 'heartbeat_back_addr': '172.21.15.169:6823/1905413450', 'heartbeat_front_addr': '172.21.15.169:6821/1905413450', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3491a731-ca96-4981-be73-e005e00c315f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6811', 'nonce': 2918557179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6813', 'nonce': 2918557179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6816', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6817', 'nonce': 2918557179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6815', 'nonce': 2918557179}]}, 'public_addr': '172.21.15.181:6811/2918557179', 'cluster_addr': '172.21.15.181:6813/2918557179', 'heartbeat_back_addr': '172.21.15.181:6817/2918557179', 'heartbeat_front_addr': '172.21.15.181:6815/2918557179', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3bdfaa34-bc10-4cec-b5c2-86e05d444ffd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6800', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6801', 'nonce': 2257685015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6803', 'nonce': 2257685015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6807', 'nonce': 2257685015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6805', 'nonce': 2257685015}]}, 'public_addr': '172.21.15.6:6801/2257685015', 'cluster_addr': '172.21.15.6:6803/2257685015', 'heartbeat_back_addr': '172.21.15.6:6807/2257685015', 'heartbeat_front_addr': '172.21.15.6:6805/2257685015', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 2399951648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 2399951648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 2399951648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 2399951648}]}, 'public_addr': '172.21.15.50:6825/2399951648', 'cluster_addr': '172.21.15.50:6827/2399951648', 'heartbeat_back_addr': '172.21.15.50:6831/2399951648', 'heartbeat_front_addr': '172.21.15.50:6829/2399951648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ddd3edea-260a-4ec9-8533-841ed933fdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6824', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6825', 'nonce': 1225429423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6826', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6827', 'nonce': 1225429423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6830', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6831', 'nonce': 1225429423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6828', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6829', 'nonce': 1225429423}]}, 'public_addr': '172.21.15.169:6825/1225429423', 'cluster_addr': '172.21.15.169:6827/1225429423', 'heartbeat_back_addr': '172.21.15.169:6831/1225429423', 'heartbeat_front_addr': '172.21.15.169:6829/1225429423', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7afc1a18-98c2-4a58-a09d-1b4f474d4dc4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6818', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6819', 'nonce': 3415295348}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6820', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6821', 'nonce': 3415295348}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6824', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6825', 'nonce': 3415295348}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6822', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6823', 'nonce': 3415295348}]}, 'public_addr': '172.21.15.181:6819/3415295348', 'cluster_addr': '172.21.15.181:6821/3415295348', 'heartbeat_back_addr': '172.21.15.181:6825/3415295348', 'heartbeat_front_addr': '172.21.15.181:6823/3415295348', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '3e45f80f-856b-4d3f-b002-255f16a14f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6824', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6825', 'nonce': 2216439459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6826', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6827', 'nonce': 2216439459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6830', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6831', 'nonce': 2216439459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6828', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6829', 'nonce': 2216439459}]}, 'public_addr': '172.21.15.6:6825/2216439459', 'cluster_addr': '172.21.15.6:6827/2216439459', 'heartbeat_back_addr': '172.21.15.6:6831/2216439459', 'heartbeat_front_addr': '172.21.15.6:6829/2216439459', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b415c423-24de-4d86-bec4-ea330364ff96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 1727176716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1727176716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1727176716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1727176716}]}, 'public_addr': '172.21.15.50:6817/1727176716', 'cluster_addr': '172.21.15.50:6819/1727176716', 'heartbeat_back_addr': '172.21.15.50:6823/1727176716', 'heartbeat_front_addr': '172.21.15.50:6821/1727176716', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e471e1cb-9c0b-4c99-8622-59eb7956770d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6808', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 3604183526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6810', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 3604183526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6814', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3604183526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6812', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3604183526}]}, 'public_addr': '172.21.15.169:6809/3604183526', 'cluster_addr': '172.21.15.169:6811/3604183526', 'heartbeat_back_addr': '172.21.15.169:6815/3604183526', 'heartbeat_front_addr': '172.21.15.169:6813/3604183526', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'c2bf4255-7ce1-4673-bedf-fd055d841fac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6826', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6827', 'nonce': 3420013159}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6828', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6829', 'nonce': 3420013159}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 3420013159}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6830', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6831', 'nonce': 3420013159}]}, 'public_addr': '172.21.15.181:6827/3420013159', 'cluster_addr': '172.21.15.181:6829/3420013159', 'heartbeat_back_addr': '172.21.15.181:6833/3420013159', 'heartbeat_front_addr': '172.21.15.181:6831/3420013159', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '4c16f3ee-f427-41d5-b0ba-9e676ddbe032', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6817', 'nonce': 2310487797}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6818', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6819', 'nonce': 2310487797}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6822', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6823', 'nonce': 2310487797}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6820', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6821', 'nonce': 2310487797}]}, 'public_addr': '172.21.15.6:6817/2310487797', 'cluster_addr': '172.21.15.6:6819/2310487797', 'heartbeat_back_addr': '172.21.15.6:6823/2310487797', 'heartbeat_front_addr': '172.21.15.6:6821/2310487797', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '0a4386aa-1a59-44c8-8d1e-7d43bf10de2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1210437211}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 1210437211}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1210437211}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 1210437211}]}, 'public_addr': '172.21.15.50:6802/1210437211', 'cluster_addr': '172.21.15.50:6806/1210437211', 'heartbeat_back_addr': '172.21.15.50:6814/1210437211', 'heartbeat_front_addr': '172.21.15.50:6810/1210437211', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '4c54e35c-e226-4d8c-bad0-51dd1cd95fe0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6800', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 4170280864}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6802', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 4170280864}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6806', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 4170280864}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6804', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 4170280864}]}, 'public_addr': '172.21.15.169:6801/4170280864', 'cluster_addr': '172.21.15.169:6803/4170280864', 'heartbeat_back_addr': '172.21.15.169:6807/4170280864', 'heartbeat_front_addr': '172.21.15.169:6805/4170280864', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '0281e750-f74a-4760-bbef-fbfa4127d06b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6803', 'nonce': 2815528414}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6805', 'nonce': 2815528414}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6809', 'nonce': 2815528414}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6807', 'nonce': 2815528414}]}, 'public_addr': '172.21.15.181:6803/2815528414', 'cluster_addr': '172.21.15.181:6805/2815528414', 'heartbeat_back_addr': '172.21.15.181:6809/2815528414', 'heartbeat_front_addr': '172.21.15.181:6807/2815528414', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.014883+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.677746+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.838957+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.321698+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.045654+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.762032+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.517939+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.664321+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.417977+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.810458+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.124128+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.554475+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.868839+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.912548+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.882601+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.090747+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 7, 'length': 1}, {'begin': 11, 'length': 1}, {'begin': 17, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:01:55.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:01:55.829 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.830 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:01:55.830 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:55.825+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:55.928 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:55.927+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:55.976 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:55.977 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:55.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:01:56.031 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:56.029+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:56.095 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:56.095 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:56.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:01:56.130 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:56.129+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:56.178 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:01:56.178 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:01:56.191 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-17T16:01:55.368813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":46170675,"num_objects":97,"num_object_clones":45,"num_object_copies":659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":4,"num_read":289,"num_read_kb":16927,"num_write":424,"num_write_kb":46514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":654,"ondisk_log_size":654,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":533852,"kb_used_data":88856,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509415588,"statfs":{"total":1546188226560,"available":1545641562112,"internally_reserved":0,"allocated":90988544,"data_stored":131727552,"data_compressed":1618026,"data_compressed_allocated":2244608,"data_compressed_original":5365760,"omap_allocated":25519,"internal_metadata":455253073},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5685248,"num_objects":11,"num_object_clones":11,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":25,"num_read_kb":5081,"num_write":43,"num_write_kb":5552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019134"},"pg_stats":[{"pgid":"3.19","version":"48'19","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.199040+0000","last_change":"2024-09-17T16:01:54.741493+0000","last_active":"2024-09-17T16:01:54.199040+0000","last_peered":"2024-09-17T16:01:54.199040+0000","last_clean":"2024-09-17T16:01:54.199040+0000","last_became_active":"2024-09-17T16:01:52.357491+0000","last_became_peered":"2024-09-17T16:01:52.357491+0000","last_unstale":"2024-09-17T16:01:54.199040+0000","last_undegraded":"2024-09-17T16:01:54.199040+0000","last_fullsized":"2024-09-17T16:01:54.199040+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:02:24.199132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2064384,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":7,"num_write_kb":2016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.18","version":"48'31","reported_seq":148,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.366201+0000","last_change":"2024-09-17T16:01:54.366201+0000","last_active":"2024-09-17T16:01:54.366201+0000","last_peered":"2024-09-17T16:01:54.366201+0000","last_clean":"2024-09-17T16:01:54.366201+0000","last_became_active":"2024-09-17T16:01:53.177881+0000","last_became_peered":"2024-09-17T16:01:53.177881+0000","last_unstale":"2024-09-17T16:01:54.366201+0000","last_undegraded":"2024-09-17T16:01:54.366201+0000","last_fullsized":"2024-09-17T16:01:54.366201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092740499999999996,"stat_sum":{"num_bytes":1892352,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":8,"num_write_kb":1496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.17","version":"52'18","reported_seq":135,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.373769+0000","last_change":"2024-09-17T16:01:54.373769+0000","last_active":"2024-09-17T16:01:54.373769+0000","last_peered":"2024-09-17T16:01:54.373769+0000","last_clean":"2024-09-17T16:01:54.373769+0000","last_became_active":"2024-09-17T16:01:52.362963+0000","last_became_peered":"2024-09-17T16:01:52.362963+0000","last_unstale":"2024-09-17T16:01:54.373769+0000","last_undegraded":"2024-09-17T16:01:54.373769+0000","last_fullsized":"2024-09-17T16:01:54.373769+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026874500000000001,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":15,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.16","version":"56'16","reported_seq":140,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.396373+0000","last_change":"2024-09-17T16:01:54.373407+0000","last_active":"2024-09-17T16:01:54.396373+0000","last_peered":"2024-09-17T16:01:54.396373+0000","last_clean":"2024-09-17T16:01:54.396373+0000","last_became_active":"2024-09-17T16:01:52.362834+0000","last_became_peered":"2024-09-17T16:01:52.362834+0000","last_unstale":"2024-09-17T16:01:54.396373+0000","last_undegraded":"2024-09-17T16:01:54.396373+0000","last_fullsized":"2024-09-17T16:01:54.396373+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023605150000000001,"stat_sum":{"num_bytes":2498560,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":909,"num_write":9,"num_write_kb":2440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.15","version":"56'8","reported_seq":132,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.492534+0000","last_change":"2024-09-17T16:01:54.746636+0000","last_active":"2024-09-17T16:01:54.492534+0000","last_peered":"2024-09-17T16:01:54.492534+0000","last_clean":"2024-09-17T16:01:54.492534+0000","last_became_active":"2024-09-17T16:01:53.142297+0000","last_became_peered":"2024-09-17T16:01:53.142297+0000","last_unstale":"2024-09-17T16:01:54.492534+0000","last_undegraded":"2024-09-17T16:01:54.492534+0000","last_fullsized":"2024-09-17T16:01:54.492534+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:02:24.492622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0064566409999999999,"stat_sum":{"num_bytes":565248,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":902,"num_write":22,"num_write_kb":552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.14","version":"48'16","reported_seq":132,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.501914+0000","last_change":"2024-09-17T16:01:53.731400+0000","last_active":"2024-09-17T16:01:53.501914+0000","last_peered":"2024-09-17T16:01:53.501914+0000","last_clean":"2024-09-17T16:01:53.501914+0000","last_became_active":"2024-09-17T16:01:52.360122+0000","last_became_peered":"2024-09-17T16:01:52.360122+0000","last_unstale":"2024-09-17T16:01:53.501914+0000","last_undegraded":"2024-09-17T16:01:53.501914+0000","last_fullsized":"2024-09-17T16:01:53.501914+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-17T16:02:23.501990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532480,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":1080,"num_write":4,"num_write_kb":520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365251+0000","last_change":"2024-09-17T16:01:20.767358+0000","last_active":"2024-09-17T16:01:54.365251+0000","last_peered":"2024-09-17T16:01:54.365251+0000","last_clean":"2024-09-17T16:01:54.365251+0000","last_became_active":"2024-09-17T16:01:18.927547+0000","last_became_peered":"2024-09-17T16:01:18.927547+0000","last_unstale":"2024-09-17T16:01:54.365251+0000","last_undegraded":"2024-09-17T16:01:54.365251+0000","last_fullsized":"2024-09-17T16:01:54.365251+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.529091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049674400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.6","version":"52'11","reported_seq":135,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.373916+0000","last_change":"2024-09-17T16:01:54.374685+0000","last_active":"2024-09-17T16:01:54.373916+0000","last_peered":"2024-09-17T16:01:54.373916+0000","last_clean":"2024-09-17T16:01:54.373916+0000","last_became_active":"2024-09-17T16:01:52.355332+0000","last_became_peered":"2024-09-17T16:01:52.355332+0000","last_unstale":"2024-09-17T16:01:54.373916+0000","last_undegraded":"2024-09-17T16:01:54.373916+0000","last_fullsized":"2024-09-17T16:01:54.373916+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032251369999999999,"stat_sum":{"num_bytes":1171456,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":907,"num_write":5,"num_write_kb":1144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,7,15,8,11,0,10],"acting":[13,7,15,8,11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365889+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:01:54.365889+0000","last_peered":"2024-09-17T16:01:54.365889+0000","last_clean":"2024-09-17T16:01:54.365889+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:01:54.365889+0000","last_undegraded":"2024-09-17T16:01:54.365889+0000","last_fullsized":"2024-09-17T16:01:54.365889+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"52'18","reported_seq":138,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.374513+0000","last_change":"2024-09-17T16:01:54.375120+0000","last_active":"2024-09-17T16:01:54.374513+0000","last_peered":"2024-09-17T16:01:54.374513+0000","last_clean":"2024-09-17T16:01:54.374513+0000","last_became_active":"2024-09-17T16:01:52.348270+0000","last_became_peered":"2024-09-17T16:01:52.348270+0000","last_unstale":"2024-09-17T16:01:54.374513+0000","last_undegraded":"2024-09-17T16:01:54.374513+0000","last_fullsized":"2024-09-17T16:01:54.374513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035050419999999999,"stat_sum":{"num_bytes":1261568,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":14,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,7,1,11,5,0],"acting":[13,6,7,1,11,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.370965+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:01:54.370965+0000","last_peered":"2024-09-17T16:01:54.370965+0000","last_clean":"2024-09-17T16:01:54.370965+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:01:54.370965+0000","last_undegraded":"2024-09-17T16:01:54.370965+0000","last_fullsized":"2024-09-17T16:01:54.370965+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:19.020054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":131,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.395605+0000","last_change":"2024-09-17T16:01:53.395522+0000","last_active":"2024-09-17T16:01:53.395605+0000","last_peered":"2024-09-17T16:01:53.395605+0000","last_clean":"2024-09-17T16:01:53.395605+0000","last_became_active":"2024-09-17T16:01:52.351079+0000","last_became_peered":"2024-09-17T16:01:52.351079+0000","last_unstale":"2024-09-17T16:01:53.395605+0000","last_undegraded":"2024-09-17T16:01:53.395605+0000","last_fullsized":"2024-09-17T16:01:53.395605+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037246099999999998,"stat_sum":{"num_bytes":532480,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":1080,"num_write":4,"num_write_kb":520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":73,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.395546+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:01:53.395546+0000","last_peered":"2024-09-17T16:01:53.395546+0000","last_clean":"2024-09-17T16:01:53.395546+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:01:53.395546+0000","last_undegraded":"2024-09-17T16:01:53.395546+0000","last_fullsized":"2024-09-17T16:01:53.395546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"35'4","reported_seq":126,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365701+0000","last_change":"2024-09-17T16:01:54.366232+0000","last_active":"2024-09-17T16:01:54.365701+0000","last_peered":"2024-09-17T16:01:54.365701+0000","last_clean":"2024-09-17T16:01:54.365701+0000","last_became_active":"2024-09-17T16:01:52.345743+0000","last_became_peered":"2024-09-17T16:01:52.345743+0000","last_unstale":"2024-09-17T16:01:54.365701+0000","last_undegraded":"2024-09-17T16:01:54.365701+0000","last_fullsized":"2024-09-17T16:01:54.365701+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071162600000000003,"stat_sum":{"num_bytes":565248,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":902,"num_write":2,"num_write_kb":552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.3","version":"54'18","reported_seq":139,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.386055+0000","last_change":"2024-09-17T16:01:54.386431+0000","last_active":"2024-09-17T16:01:54.386055+0000","last_peered":"2024-09-17T16:01:54.386055+0000","last_clean":"2024-09-17T16:01:54.386055+0000","last_became_active":"2024-09-17T16:01:52.349931+0000","last_became_peered":"2024-09-17T16:01:52.349931+0000","last_unstale":"2024-09-17T16:01:54.386055+0000","last_undegraded":"2024-09-17T16:01:54.386055+0000","last_fullsized":"2024-09-17T16:01:54.386055+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064565200000000003,"stat_sum":{"num_bytes":1400832,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":4,"num_read_kb":1,"num_write":5,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":78,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365158+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:01:54.365158+0000","last_peered":"2024-09-17T16:01:54.365158+0000","last_clean":"2024-09-17T16:01:54.365158+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:01:54.365158+0000","last_undegraded":"2024-09-17T16:01:54.365158+0000","last_fullsized":"2024-09-17T16:01:54.365158+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"48'27","reported_seq":140,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.366954+0000","last_change":"2024-09-17T16:01:53.366885+0000","last_active":"2024-09-17T16:01:53.366954+0000","last_peered":"2024-09-17T16:01:53.366954+0000","last_clean":"2024-09-17T16:01:53.366954+0000","last_became_active":"2024-09-17T16:01:52.345493+0000","last_became_peered":"2024-09-17T16:01:52.345493+0000","last_unstale":"2024-09-17T16:01:53.366954+0000","last_undegraded":"2024-09-17T16:01:53.366954+0000","last_fullsized":"2024-09-17T16:01:53.366954+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001122355,"stat_sum":{"num_bytes":1056768,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":67,"num_write":15,"num_write_kb":1032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.385249+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:01:54.385249+0000","last_peered":"2024-09-17T16:01:54.385249+0000","last_clean":"2024-09-17T16:01:54.385249+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:01:54.385249+0000","last_undegraded":"2024-09-17T16:01:54.385249+0000","last_fullsized":"2024-09-17T16:01:54.385249+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"52'11","reported_seq":131,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.372015+0000","last_change":"2024-09-17T16:01:54.372973+0000","last_active":"2024-09-17T16:01:54.372015+0000","last_peered":"2024-09-17T16:01:54.372015+0000","last_clean":"2024-09-17T16:01:54.372015+0000","last_became_active":"2024-09-17T16:01:52.354030+0000","last_became_peered":"2024-09-17T16:01:52.354030+0000","last_unstale":"2024-09-17T16:01:54.372015+0000","last_undegraded":"2024-09-17T16:01:54.372015+0000","last_fullsized":"2024-09-17T16:01:54.372015+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022459139999999999,"stat_sum":{"num_bytes":1228800,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.364927+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:01:54.364927+0000","last_peered":"2024-09-17T16:01:54.364927+0000","last_clean":"2024-09-17T16:01:54.364927+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:01:54.364927+0000","last_undegraded":"2024-09-17T16:01:54.364927+0000","last_fullsized":"2024-09-17T16:01:54.364927+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"48'26","reported_seq":152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.366582+0000","last_change":"2024-09-17T16:01:54.367320+0000","last_active":"2024-09-17T16:01:54.366582+0000","last_peered":"2024-09-17T16:01:54.366582+0000","last_clean":"2024-09-17T16:01:54.366582+0000","last_became_active":"2024-09-17T16:01:52.349490+0000","last_became_peered":"2024-09-17T16:01:52.349490+0000","last_unstale":"2024-09-17T16:01:54.366582+0000","last_undegraded":"2024-09-17T16:01:54.366582+0000","last_fullsized":"2024-09-17T16:01:54.366582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015316570000000001,"stat_sum":{"num_bytes":1556480,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1031,"num_write":7,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":77,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365502+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:01:54.365502+0000","last_peered":"2024-09-17T16:01:54.365502+0000","last_clean":"2024-09-17T16:01:54.365502+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:01:54.365502+0000","last_undegraded":"2024-09-17T16:01:54.365502+0000","last_fullsized":"2024-09-17T16:01:54.365502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":211,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.365609+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:01:54.365609+0000","last_peered":"2024-09-17T16:01:54.365609+0000","last_clean":"2024-09-17T16:01:54.365609+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:01:54.365609+0000","last_undegraded":"2024-09-17T16:01:54.365609+0000","last_fullsized":"2024-09-17T16:01:54.365609+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"48'31","reported_seq":148,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.366268+0000","last_change":"2024-09-17T16:01:54.367020+0000","last_active":"2024-09-17T16:01:54.366268+0000","last_peered":"2024-09-17T16:01:54.366268+0000","last_clean":"2024-09-17T16:01:54.366268+0000","last_became_active":"2024-09-17T16:01:52.347600+0000","last_became_peered":"2024-09-17T16:01:52.347600+0000","last_unstale":"2024-09-17T16:01:54.366268+0000","last_undegraded":"2024-09-17T16:01:54.366268+0000","last_fullsized":"2024-09-17T16:01:54.366268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00154185,"stat_sum":{"num_bytes":1892352,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.9","version":"48'19","reported_seq":133,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.370002+0000","last_change":"2024-09-17T16:01:52.347990+0000","last_active":"2024-09-17T16:01:53.370002+0000","last_peered":"2024-09-17T16:01:53.370002+0000","last_clean":"2024-09-17T16:01:53.370002+0000","last_became_active":"2024-09-17T16:01:52.347076+0000","last_became_peered":"2024-09-17T16:01:52.347076+0000","last_unstale":"2024-09-17T16:01:53.370002+0000","last_undegraded":"2024-09-17T16:01:53.370002+0000","last_fullsized":"2024-09-17T16:01:53.370002+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000909552,"stat_sum":{"num_bytes":2064384,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":585,"num_write":6,"num_write_kb":2016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.a","version":"52'32","reported_seq":120,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-17T16:01:51.364601+0000","last_change":"2024-09-17T16:01:50.381423+0000","last_active":"2024-09-17T16:01:51.364601+0000","last_peered":"2024-09-17T16:01:51.364601+0000","last_clean":"2024-09-17T16:01:51.364601+0000","last_became_active":"2024-09-17T16:01:32.579608+0000","last_became_peered":"2024-09-17T16:01:32.579608+0000","last_unstale":"2024-09-17T16:01:51.364601+0000","last_undegraded":"2024-09-17T16:01:51.364601+0000","last_fullsized":"2024-09-17T16:01:51.364601+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:45.699356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007019338,"stat_sum":{"num_bytes":3293184,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":14,"num_write_kb":3216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,7,3,14,6,8],"acting":[12,4,7,3,14,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.b","version":"48'16","reported_seq":127,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.370631+0000","last_change":"2024-09-17T16:01:54.372459+0000","last_active":"2024-09-17T16:01:54.370631+0000","last_peered":"2024-09-17T16:01:54.370631+0000","last_clean":"2024-09-17T16:01:54.370631+0000","last_became_active":"2024-09-17T16:01:32.027225+0000","last_became_peered":"2024-09-17T16:01:32.027225+0000","last_unstale":"2024-09-17T16:01:54.370631+0000","last_undegraded":"2024-09-17T16:01:54.370631+0000","last_fullsized":"2024-09-17T16:01:54.370631+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:31.964453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021097249999999998,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":14,"num_read_kb":857,"num_write":9,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,7],"acting":[13,1,11,15,6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.c","version":"48'20","reported_seq":125,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.382272+0000","last_change":"2024-09-17T16:01:54.382716+0000","last_active":"2024-09-17T16:01:54.382272+0000","last_peered":"2024-09-17T16:01:54.382272+0000","last_clean":"2024-09-17T16:01:54.382272+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:01:54.382272+0000","last_undegraded":"2024-09-17T16:01:54.382272+0000","last_fullsized":"2024-09-17T16:01:54.382272+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.948112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075196900000000001,"stat_sum":{"num_bytes":1933312,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":2531,"num_write":9,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.d","version":"52'41","reported_seq":145,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-17T16:01:51.364530+0000","last_change":"2024-09-17T16:01:50.384011+0000","last_active":"2024-09-17T16:01:51.364530+0000","last_peered":"2024-09-17T16:01:51.364530+0000","last_clean":"2024-09-17T16:01:51.364530+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:01:51.364530+0000","last_undegraded":"2024-09-17T16:01:51.364530+0000","last_fullsized":"2024-09-17T16:01:51.364530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:56.255053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0095653089999999993,"stat_sum":{"num_bytes":5062656,"num_objects":10,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":2059,"num_write":25,"num_write_kb":5232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"3","length":"1"},{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.e","version":"52'35","reported_seq":139,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.369705+0000","last_change":"2024-09-17T16:01:50.355604+0000","last_active":"2024-09-17T16:01:53.369705+0000","last_peered":"2024-09-17T16:01:53.369705+0000","last_clean":"2024-09-17T16:01:53.369705+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:01:53.369705+0000","last_undegraded":"2024-09-17T16:01:53.369705+0000","last_fullsized":"2024-09-17T16:01:53.369705+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.276841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088201930000000005,"stat_sum":{"num_bytes":5996544,"num_objects":9,"num_object_clones":5,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":16,"num_read_kb":2456,"num_write":41,"num_write_kb":5856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.f","version":"54'24","reported_seq":129,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.382592+0000","last_change":"2024-09-17T16:01:54.382863+0000","last_active":"2024-09-17T16:01:54.382592+0000","last_peered":"2024-09-17T16:01:54.382592+0000","last_clean":"2024-09-17T16:01:54.382592+0000","last_became_active":"2024-09-17T16:01:31.980473+0000","last_became_peered":"2024-09-17T16:01:31.980473+0000","last_unstale":"2024-09-17T16:01:54.382592+0000","last_undegraded":"2024-09-17T16:01:54.382592+0000","last_fullsized":"2024-09-17T16:01:54.382592+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:50.639200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000531991,"stat_sum":{"num_bytes":1064960,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":12,"num_read_kb":367,"num_write":11,"num_write_kb":864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,7,10,1],"acting":[0,2,14,11,7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.10","version":"55'29","reported_seq":139,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-17T16:01:53.376274+0000","last_change":"2024-09-17T16:01:52.356559+0000","last_active":"2024-09-17T16:01:53.376274+0000","last_peered":"2024-09-17T16:01:53.376274+0000","last_clean":"2024-09-17T16:01:53.376274+0000","last_became_active":"2024-09-17T16:01:52.355786+0000","last_became_peered":"2024-09-17T16:01:52.355786+0000","last_unstale":"2024-09-17T16:01:53.376274+0000","last_undegraded":"2024-09-17T16:01:53.376274+0000","last_fullsized":"2024-09-17T16:01:53.376274+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1368064,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":68,"num_write":17,"num_write_kb":1336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"}]},{"pgid":"3.11","version":"52'11","reported_seq":128,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.374427+0000","last_change":"2024-09-17T16:01:54.374427+0000","last_active":"2024-09-17T16:01:54.374427+0000","last_peered":"2024-09-17T16:01:54.374427+0000","last_clean":"2024-09-17T16:01:54.374427+0000","last_became_active":"2024-09-17T16:01:52.748295+0000","last_became_peered":"2024-09-17T16:01:52.748295+0000","last_unstale":"2024-09-17T16:01:54.374427+0000","last_undegraded":"2024-09-17T16:01:54.374427+0000","last_fullsized":"2024-09-17T16:01:54.374427+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030364939999999998,"stat_sum":{"num_bytes":1228800,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,7],"acting":[13,4,3,11,10,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.12","version":"48'26","reported_seq":152,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.366476+0000","last_change":"2024-09-17T16:01:54.366476+0000","last_active":"2024-09-17T16:01:54.366476+0000","last_peered":"2024-09-17T16:01:54.366476+0000","last_clean":"2024-09-17T16:01:54.366476+0000","last_became_active":"2024-09-17T16:01:53.002358+0000","last_became_peered":"2024-09-17T16:01:53.002358+0000","last_unstale":"2024-09-17T16:01:54.366476+0000","last_undegraded":"2024-09-17T16:01:54.366476+0000","last_fullsized":"2024-09-17T16:01:54.366476+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000999303,"stat_sum":{"num_bytes":1556480,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":1031,"num_write":7,"num_write_kb":1480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,7],"acting":[11,2,0,4,8,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]},{"pgid":"3.13","version":"52'16","reported_seq":138,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T16:01:54.386134+0000","last_change":"2024-09-17T16:01:54.386134+0000","last_active":"2024-09-17T16:01:54.386134+0000","last_peered":"2024-09-17T16:01:54.386134+0000","last_clean":"2024-09-17T16:01:54.386134+0000","last_became_active":"2024-09-17T16:01:52.366019+0000","last_became_peered":"2024-09-17T16:01:52.366019+0000","last_unstale":"2024-09-17T16:01:54.386134+0000","last_undegraded":"2024-09-17T16:01:54.386134+0000","last_fullsized":"2024-09-17T16:01:54.386134+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069486099999999996,"stat_sum":{"num_bytes":1400832,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":1368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"7","length":"1"},{"start":"b","length":"1"},{"start":"11","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":45580288,"num_objects":93,"num_object_clones":45,"num_object_copies":651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":4,"num_read":203,"num_read_kb":16854,"num_write":277,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":82534400,"data_stored":127344640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":539,"ondisk_log_size":539,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247819,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30788,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341052,"statfs":{"total":96636764160,"available":96605237248,"internally_reserved":0,"allocated":3850240,"data_stored":5938734,"data_compressed":93336,"data_compressed_allocated":126976,"data_compressed_original":253952,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280523,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31164,"kb_used_data":4128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340676,"statfs":{"total":96636764160,"available":96604852224,"internally_reserved":0,"allocated":4227072,"data_stored":5312184,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":5924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":6066176,"data_stored":9133752,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313228,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35316,"kb_used_data":4128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336524,"statfs":{"total":96636764160,"available":96600600576,"internally_reserved":0,"allocated":4227072,"data_stored":5934776,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280523,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35096,"kb_used_data":8068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336744,"statfs":{"total":96636764160,"available":96600825856,"internally_reserved":0,"allocated":8261632,"data_stored":11935416,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280523,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39296,"kb_used_data":8108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332544,"statfs":{"total":96636764160,"available":96596525056,"internally_reserved":0,"allocated":8302592,"data_stored":12095160,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":5940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":6082560,"data_stored":8630488,"data_compressed":109931,"data_compressed_allocated":172032,"data_compressed_original":782336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30616,"kb_used_data":3584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341224,"statfs":{"total":96636764160,"available":96605413376,"internally_reserved":0,"allocated":3670016,"data_stored":5119672,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,7,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280523,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39920,"kb_used_data":8732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331920,"statfs":{"total":96636764160,"available":96595886080,"internally_reserved":0,"allocated":8941568,"data_stored":12861112,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32548,"kb_used_data":5512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339292,"statfs":{"total":96636764160,"available":96603435008,"internally_reserved":0,"allocated":5644288,"data_stored":8564408,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280523,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":5248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":5373952,"data_stored":9049873,"data_compressed":102485,"data_compressed_allocated":139264,"data_compressed_original":278528,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,7,8,9,10,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280523,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":4861952,"data_stored":7522065,"data_compressed":102485,"data_compressed_allocated":139264,"data_compressed_original":278528,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,6,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280523,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":5980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":6123520,"data_stored":9148196,"data_compressed":102485,"data_compressed_allocated":139264,"data_compressed_original":278528,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30236,"kb_used_data":3200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341604,"statfs":{"total":96636764160,"available":96605802496,"internally_reserved":0,"allocated":3276800,"data_stored":3727051,"data_compressed":99432,"data_compressed_allocated":135168,"data_compressed_original":270336,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280523,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30824,"kb_used_data":3792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341016,"statfs":{"total":96636764160,"available":96605200384,"internally_reserved":0,"allocated":3883008,"data_stored":4712209,"data_compressed":102485,"data_compressed_allocated":139264,"data_compressed_original":278528,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,3,4,5,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280522,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35032,"kb_used_data":8004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336808,"statfs":{"total":96636764160,"available":96600891392,"internally_reserved":0,"allocated":8196096,"data_stored":12042456,"data_compressed":109931,"data_compressed_allocated":172032,"data_compressed_original":782336,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,6,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7991296,"data_stored":12300288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3624960,"data_stored":5537792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5423104,"data_stored":7704576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3067904,"data_stored":4722688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8339456,"data_stored":12464128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5042176,"data_stored":8167424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4759552,"data_stored":8642560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4247552,"data_stored":7114752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":5505024,"data_stored":8740864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2670592,"data_stored":3330048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3268608,"data_stored":4304896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":7868416,"data_stored":11878400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":3272704,"data_stored":5562368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":8032256,"data_stored":12460032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5464064,"data_stored":8736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3956736,"data_stored":5677056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:01:56.193 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [13, 15] 2024-09-17T16:01:56.193 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.7', '13', '15'] 2024-09-17T16:01:56.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.7 13 15 2024-09-17T16:01:56.232 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:56.230+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:56.253 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:56.253 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:56.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:01:56.333 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:56.331+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:56.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:01:56.377 INFO:tasks.rados.rados.0.smithi181.stdout:153: done (3 left) 2024-09-17T16:01:56.377 INFO:tasks.rados.rados.0.smithi181.stdout:154: snap_create 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 2 to smithi18136759-33 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 28 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 3 to smithi18136759-33 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 4 to smithi18136759-33 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: finishing write tid 5 to smithi18136759-33 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 33 updating version 0 to 21 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 33 version 21 is already newer than 20 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 33 updating version 21 to 22 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: oid 33 updating version 22 to 23 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 23 (ObjNum 79 snap 17 seq_num 79) dirty exists 2024-09-17T16:01:56.378 INFO:tasks.rados.rados.0.smithi181.stdout:150: left oid 33 (ObjNum 79 snap 17 seq_num 79) 2024-09-17T16:01:56.420 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:56.420 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:56.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:01:56.474 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:01:56.472+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:01:56.497 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:56.497 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:56.520 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:56.520 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:56.534 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:56.532+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:56.539 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:56.539 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:56.541 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:56.542 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:56.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:01:56.564 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:56.564 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:56.584 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:56.584 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:56.606 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:56.606 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:56.628 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:56.628 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:56.635 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:01:56.633+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:01:56.654 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:56.655 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:56.669 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:56.669 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:56.669 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:56.669 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:56.675 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:56.675 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:56.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:01:56.696 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:56.696 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:56.717 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:56.717 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:56.736 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:01:56.734+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:01:56.739 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:56.739 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:56.759 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:56.759 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:56.780 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:56.778+0000 7f700d41b640 0 -- 172.21.15.6:0/4184163622 >> [v2:172.21.15.50:6800/1210437211,v1:172.21.15.50:6802/1210437211] conn(0x7f700821bd90 msgr2=0x7f700821e180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:56.786 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:56.786 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:56.787 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:56.787 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:56.787 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:56.787 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:56.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:01:56.808 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:56.808 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:56.830 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:56.830 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:56.839 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:01:56.837+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 44/32 ECSubWrite(tid=169, reqid=client.4560.0:229, at_version=44'23, trim_to=0'0, roll_forward_to=39'20))", 2024-09-17T16:01:56.904 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.918475+0000", 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.982661803999999, 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001866993, 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918475+0000", 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918475+0000", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918475+0000", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.18e-07 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918907+0000", 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043166199999999997 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918927+0000", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0251999999999999e-05 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918934+0000", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5889999999999999e-06 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918989+0000", 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4700000000000001e-05 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.918995+0000", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3160000000000002e-06 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.920299+0000", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013045999999999999 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.920342+0000", 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2756000000000002e-05 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:238 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.979822202999999, 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011165420000000001, 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.909 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 238 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921316+0000", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3480000000000001e-06 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921322+0000", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921333+0000", 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0779e-05 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921339+0000", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9699999999999996e-06 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921387+0000", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8588000000000003e-05 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921462+0000", 2024-09-17T16:01:56.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4530999999999995e-05 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922070+0000", 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060823699999999997 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923628+0000", 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001558219 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932393+0000", 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087645399999999995 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932480+0000", 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7435000000000006e-05 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:239 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 5] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:01:56.914 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.979782623, 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011291409000000001, 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 239 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921355+0000", 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921358+0000", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.568e-06 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921365+0000", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8040000000000004e-06 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921371+0000", 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.536e-06 2024-09-17T16:01:56.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922590+0000", 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012196679999999999 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922675+0000", 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4715000000000005e-05 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923334+0000", 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065929300000000003 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925081+0000", 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001747024 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932583+0000", 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075011599999999998 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932646+0000", 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3200999999999993e-05 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:240 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.979744413000001, 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011490029000000001, 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.920 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 240 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7899999999999999e-07 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921396+0000", 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.52e-06 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921402+0000", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309999999999997e-06 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921408+0000", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5690000000000004e-06 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923661+0000", 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022530979999999998 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923874+0000", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021281499999999999 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924326+0000", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045228500000000002 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925665+0000", 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013387049999999999 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932831+0000", 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071659510000000003 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932883+0000", 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1975999999999998e-05 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.925 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 44/32 ECSubWrite(tid=171, reqid=client.4560.0:231, at_version=44'25, trim_to=0'0, roll_forward_to=39'20))", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.922379+0000", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.978758138, 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001442032, 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922379+0000", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922379+0000", 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922380+0000", 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4700000000000001e-07 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922394+0000", 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4531e-05 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922422+0000", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7678000000000002e-05 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922429+0000", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922481+0000", 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1931999999999997e-05 2024-09-17T16:01:56.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922490+0000", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9239999999999996e-06 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923789+0000", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012987210000000001 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923821+0000", 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2270000000000001e-05 2024-09-17T16:01:56.929 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs1 44/32 ECSubWrite(tid=58, reqid=client.4560.0:256, at_version=44'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.929912+0000", 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.971224569, 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002690502, 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929912+0000", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929912+0000", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929913+0000", 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-07 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929946+0000", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2190000000000002e-05 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929965+0000", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9536000000000001e-05 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929982+0000", 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6398000000000001e-05 2024-09-17T16:01:56.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930045+0000", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3830000000000004e-05 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930056+0000", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0433e-05 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932567+0000", 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025107800000000002 2024-09-17T16:01:56.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932603+0000", 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6355e-05 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 44/32 ECSubWrite(tid=34, reqid=client.4560.0:248, at_version=44'12, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.940712+0000", 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.960425003999999, 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016980389999999999, 2024-09-17T16:01:56.934 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940712+0000", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940712+0000", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940713+0000", 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8999999999999996e-07 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940903+0000", 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000189855 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940934+0000", 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1353000000000003e-05 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940940+0000", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.940992+0000", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2082e-05 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.941003+0000", 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1547e-05 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.942372+0000", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013688929999999999 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.942410+0000", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7953000000000001e-05 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:272 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:01:56.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.975456513999999, 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046207150000000001, 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 272 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:01:56.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925682+0000", 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7060000000000001e-06 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925690+0000", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3950000000000003e-06 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925706+0000", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.611e-05 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925713+0000", 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9809999999999997e-06 2024-09-17T16:01:56.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2625000000000003e-05 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925900+0000", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011498 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926711+0000", 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081025800000000001 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929013+0000", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023021669999999999 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930254+0000", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012415900000000001 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930301+0000", 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6903000000000001e-05 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.943 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:273 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 8] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.975407596, 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070115810000000002, 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.944 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 273 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925730+0000", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925733+0000", 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3900000000000002e-06 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925740+0000", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6610000000000003e-06 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925752+0000", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2038000000000001e-05 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927223+0000", 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014708670000000001 2024-09-17T16:01:56.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927310+0000", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7535999999999997e-05 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927853+0000", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054293600000000005 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929016+0000", 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001162445 2024-09-17T16:01:56.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932681+0000", 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003665092 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932741+0000", 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0327e-05 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.949 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:274 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.975357214000001, 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011049606999999999, 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 274 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.84e-07 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.425e-06 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925790+0000", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2100000000000001e-06 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925793+0000", 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3409999999999999e-06 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928169+0000", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023761490000000001 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928252+0000", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2699000000000002e-05 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928677+0000", 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042531099999999999 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929281+0000", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060397100000000002 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936793+0000", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00751164 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936829+0000", 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6477000000000003e-05 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.954 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs1 46/32 ECSubWrite(tid=67, reqid=osd.13.0:67, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.916796+0000", 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.984340727999999, 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014490970000000001, 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916796+0000", 2024-09-17T16:01:56.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916796+0000", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916797+0000", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916822+0000", 2024-09-17T16:01:56.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4927e-05 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916877+0000", 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4812999999999998e-05 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916886+0000", 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1009999999999998e-06 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916955+0000", 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8594000000000001e-05 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916965+0000", 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.027e-05 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918191+0000", 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012258130000000001 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918245+0000", 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4789e-05 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s3 46/32 ECSubWrite(tid=68, reqid=osd.13.0:68, at_version=46'12, trim_to=0'0, roll_forward_to=42'10))", 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.916984+0000", 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.984152804000001, 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019200179999999999, 2024-09-17T16:01:56.959 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916984+0000", 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916984+0000", 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.916985+0000", 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3399999999999997e-07 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917007+0000", 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1379e-05 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917045+0000", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8219000000000001e-05 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917053+0000", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5520000000000005e-06 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917121+0000", 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8155000000000001e-05 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917128+0000", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3260000000000001e-06 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918875+0000", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017469989999999999 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918904+0000", 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9454e-05 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.963 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 46/32 ECSubWrite(tid=16, reqid=osd.15.0:16, at_version=46'12, trim_to=0'0, roll_forward_to=39'10))", 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.917661+0000", 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.983476261, 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020084130000000001, 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917661+0000", 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917661+0000", 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917661+0000", 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6e-07 2024-09-17T16:01:56.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917684+0000", 2024-09-17T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3365000000000001e-05 2024-09-17T16:01:56.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917750+0000", 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6032999999999996e-05 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917763+0000", 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2311e-05 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917862+0000", 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9350000000000003e-05 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917875+0000", 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2642e-05 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919626+0000", 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017510819999999999 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919669+0000", 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.337e-05 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.8s1 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.921293+0000", 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.979843888, 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001490913, 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.969 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921293+0000", 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921293+0000", 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.970 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921294+0000", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921310+0000", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6390999999999999e-05 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921335+0000", 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.491e-05 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6849999999999998e-06 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921396+0000", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4830999999999997e-05 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.921406+0000", 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0169999999999999e-05 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.922751+0000", 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013452340000000001 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.922784+0000", 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3018000000000001e-05 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:56.973 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925039+0000", 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.976097887, 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018967630000000001, 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925039+0000", 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925039+0000", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925039+0000", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6399999999999998e-07 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925059+0000", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9519999999999999e-05 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925136+0000", 2024-09-17T16:01:56.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7428000000000005e-05 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925143+0000", 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6889999999999998e-06 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925574+0000", 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043066300000000002 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925583+0000", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3829999999999998e-06 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926906+0000", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013223390000000001 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926936+0000", 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0377000000000001e-05 2024-09-17T16:01:56.977 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.bs1 46/32 ECSubWrite(tid=70, reqid=osd.13.0:67, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925373+0000", 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.975764301, 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001522233, 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925373+0000", 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925373+0000", 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925374+0000", 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2999999999999999e-07 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925385+0000", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1474000000000001e-05 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925420+0000", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5138999999999997e-05 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925431+0000", 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0881e-05 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925498+0000", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6910999999999997e-05 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925508+0000", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9389999999999992e-06 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926859+0000", 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001350749 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926895+0000", 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6310000000000003e-05 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s3 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351139+0000", 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5499976780000004, 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001568533, 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.982 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351139+0000", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351139+0000", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351140+0000", 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.16e-07 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351169+0000", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9291000000000001e-05 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351579+0000", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041033900000000002 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351588+0000", 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3310000000000004e-06 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351666+0000", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8973000000000004e-05 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351676+0000", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6169999999999996e-06 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.352693+0000", 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010165319999999999 2024-09-17T16:01:56.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.352708+0000", 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5234e-05 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/32 ECSubWrite(tid=53, reqid=client.4560.0:344, at_version=54'24, trim_to=0'0, roll_forward_to=52'22))", 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.558888144, 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036631329999999998, 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.986 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8099999999999998e-07 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342277+0000", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7398000000000001e-05 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342817+0000", 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054081699999999997 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342835+0000", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342920+0000", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4445999999999998e-05 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342932+0000", 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2283000000000001e-05 2024-09-17T16:01:56.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345862+0000", 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002929577 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345912+0000", 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0451000000000002e-05 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/32 ECSubWrite(tid=54, reqid=client.4560.0:344, at_version=0'0, trim_to=0'0, roll_forward_to=54'24))", 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.348486+0000", 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5526505369999999, 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018387379999999999, 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348486+0000", 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348486+0000", 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348487+0000", 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9800000000000002e-07 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348501+0000", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3599e-05 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348544+0000", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3294999999999998e-05 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348629+0000", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5147000000000003e-05 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348697+0000", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7786000000000005e-05 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.348712+0000", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4329e-05 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350291+0000", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001579235 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350325+0000", 2024-09-17T16:01:56.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4449000000000002e-05 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 54/53 ECSubWrite(tid=23, reqid=client.4560.0:346, at_version=54'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.359470+0000", 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5416674879999999, 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014126290000000001, 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:56.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359470+0000", 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359470+0000", 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359470+0000", 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9700000000000005e-07 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359514+0000", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3778e-05 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359550+0000", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6198000000000001e-05 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359560+0000", 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7520000000000008e-06 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359627+0000", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6829999999999995e-05 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.359647+0000", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0194e-05 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.360819+0000", 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011717629999999999 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:56.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.360882+0000", 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3217000000000001e-05 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:56.999 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:01:57.001 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:56.998+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:57.101 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:57.100+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:57.149 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.150 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:57.150 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:57.150 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:01:57.202 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:57.200+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:57.265 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.265 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:57.265 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:57.265 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:57.265 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:01:57.303 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:01:57.301+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:01:57.334 INFO:teuthology.orchestra.run.smithi006.stderr:set 2.7 pg_upmap mapping to [13,15] 2024-09-17T16:01:57.337 INFO:tasks.rados.rados.0.smithi181.stdout:150: done (3 left) 2024-09-17T16:01:57.337 INFO:tasks.rados.rados.0.smithi181.stdout:151: done (2 left) 2024-09-17T16:01:57.337 INFO:tasks.rados.rados.0.smithi181.stdout:152: done (1 left) 2024-09-17T16:01:57.337 INFO:tasks.rados.rados.0.smithi181.stdout:154: done (0 left) 2024-09-17T16:01:57.337 INFO:tasks.rados.rados.0.smithi181.stdout:155: snap_remove snap 17 2024-09-17T16:01:57.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=33, reqid=client.4560.0:233, at_version=44'9, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.501425757, 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136534, 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.427 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922734+0000", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922735+0000", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1340000000000001e-06 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922924+0000", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018932500000000001 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922965+0000", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0547000000000003e-05 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922974+0000", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4080000000000004e-06 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923042+0000", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7964000000000002e-05 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923056+0000", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4018e-05 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924052+0000", 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000996432 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924099+0000", 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6511999999999998e-05 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:57.431 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.495524976, 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:01:57.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:01:57.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=35, reqid=client.4560.0:242, at_version=44'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.493821558, 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023531030000000001, 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930339+0000", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1909999999999999e-06 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931084+0000", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074480300000000004 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931122+0000", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8195000000000003e-05 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931132+0000", 2024-09-17T16:01:57.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931291+0000", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015850099999999999 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931302+0000", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1487e-05 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932657+0000", 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001355001 2024-09-17T16:01:57.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932691+0000", 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3426000000000002e-05 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=37, reqid=client.4560.0:236, at_version=44'15, trim_to=0'0, roll_forward_to=44'11))", 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.488077287999999, 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015795060000000001, 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.440 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936082+0000", 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936083+0000", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.279e-06 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936265+0000", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018191799999999999 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936310+0000", 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5139000000000003e-05 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936321+0000", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0631e-05 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936421+0000", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9679000000000006e-05 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.936430+0000", 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1379999999999996e-06 2024-09-17T16:01:57.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937627+0000", 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011968930000000001 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.937662+0000", 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4829000000000001e-05 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.504855985000001, 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026452619999999998, 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.445 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.14e-07 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919991+0000", 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068760399999999997 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.446 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920025+0000", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4292000000000003e-05 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920043+0000", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8028000000000001e-05 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920118+0000", 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4583000000000001e-05 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.447 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920129+0000", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1105e-05 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921921+0000", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017915590000000001 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921949+0000", 2024-09-17T16:01:57.448 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7977e-05 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 46/32 ECSubWrite(tid=68, reqid=osd.13.0:68, at_version=46'12, trim_to=0'0, roll_forward_to=42'10))", 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.507061464, 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015030169999999999, 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.449 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917098+0000", 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:01:57.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917127+0000", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9088999999999999e-05 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917562+0000", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043494999999999997 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917573+0000", 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0749e-05 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.451 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917670+0000", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7249000000000004e-05 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917681+0000", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1224e-05 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918552+0000", 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00087075200000000003 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.452 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918601+0000", 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8752999999999999e-05 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 46/32 ECSubWrite(tid=30, reqid=osd.2.0:30, at_version=46'18, trim_to=0'0, roll_forward_to=45'16))", 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.506533908, 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018538020000000001, 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.453 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917625+0000", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917626+0000", 2024-09-17T16:01:57.454 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9600000000000001e-07 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917649+0000", 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3079e-05 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917683+0000", 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4134999999999998e-05 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.455 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917691+0000", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9840000000000001e-06 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917767+0000", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6228000000000003e-05 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.917778+0000", 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0507e-05 2024-09-17T16:01:57.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919440+0000", 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016621909999999999 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919479+0000", 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9481999999999999e-05 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.505532026000001, 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019496909999999999, 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918627+0000", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918628+0000", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3699999999999999e-07 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918653+0000", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5582999999999999e-05 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918733+0000", 2024-09-17T16:01:57.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0156999999999998e-05 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918742+0000", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5580000000000001e-06 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918812+0000", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0308999999999994e-05 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918823+0000", 2024-09-17T16:01:57.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0370999999999999e-05 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920536+0000", 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001713525 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920577+0000", 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0951000000000001e-05 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.504608592, 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013059860000000001, 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919551+0000", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919552+0000", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919575+0000", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3499e-05 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919611+0000", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5547999999999999e-05 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919620+0000", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3659999999999998e-06 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919715+0000", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4709000000000004e-05 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0769999999999997e-06 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920812+0000", 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001088049 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920857+0000", 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4922999999999997e-05 2024-09-17T16:01:57.465 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.498773160000001, 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.466 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.491947544, 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:01:57.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 52/32 ECSubWrite(tid=47, reqid=osd.2.0:47, at_version=52'33, trim_to=0'0, roll_forward_to=51'31))", 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.347697+0000", 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.0764621590000001, 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014480649999999999, 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.475 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347697+0000", 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347697+0000", 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347697+0000", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6899999999999999e-07 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347732+0000", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347785+0000", 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2540000000000002e-05 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347797+0000", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2704000000000001e-05 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347885+0000", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7684999999999998e-05 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.347898+0000", 2024-09-17T16:01:57.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2777e-05 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.349109+0000", 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001211057 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.349145+0000", 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6338999999999999e-05 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 52/32 ECSubWrite(tid=48, reqid=osd.2.0:48, at_version=52'35, trim_to=0'0, roll_forward_to=51'31))", 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.348732+0000", 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.075427275, 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001314866, 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348732+0000", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348732+0000", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348733+0000", 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1699999999999998e-07 2024-09-17T16:01:57.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348747+0000", 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4481000000000001e-05 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348772+0000", 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.529e-05 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348778+0000", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3469999999999998e-06 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348833+0000", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5386999999999997e-05 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.348851+0000", 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7499e-05 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.350009+0000", 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011579559999999999 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.350047+0000", 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8288999999999999e-05 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 7.0726231359999998, 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017347922000000002, 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351537+0000", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351558+0000", 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1399e-05 2024-09-17T16:01:57.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367847+0000", 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289428000000002 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367858+0000", 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0835e-05 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367938+0000", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0104000000000004e-05 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367950+0000", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1153e-05 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368846+0000", 2024-09-17T16:01:57.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089654899999999996 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368884+0000", 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8075999999999999e-05 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0739129260000002, 2024-09-17T16:01:57.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020622370000000002, 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350247+0000", 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.67e-07 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350271+0000", 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3926000000000001e-05 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350316+0000", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5837000000000002e-05 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350327+0000", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0597e-05 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350399+0000", 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2203e-05 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350412+0000", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2364999999999999e-05 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.352262+0000", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018508299999999999 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.352309+0000", 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6312e-05 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.493 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s4 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.051267255, 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026391370000000002, 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.29e-07 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373128+0000", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023566099999999999 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373174+0000", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6307999999999998e-05 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373186+0000", 2024-09-17T16:01:57.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2084e-05 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373282+0000", 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5749999999999996e-05 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373296+0000", 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3468999999999999e-05 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375480+0000", 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002184348 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375531+0000", 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1388000000000003e-05 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=97, reqid=client.4560.0:353, at_version=55'45, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.374811+0000", 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0493484239999997, 2024-09-17T16:01:57.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001884284, 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.374811+0000", 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.374811+0000", 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.374812+0000", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8700000000000005e-07 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375165+0000", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035372600000000001 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375207+0000", 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1451e-05 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375217+0000", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0213e-05 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375295+0000", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8041999999999996e-05 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375306+0000", 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1269e-05 2024-09-17T16:01:57.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376651+0000", 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001344308 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376695+0000", 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4487999999999998e-05 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=98, reqid=client.4560.0:354, at_version=55'46, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:01:57.502 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.377411+0000", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.04674814, 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001397269, 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377411+0000", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377411+0000", 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377412+0000", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7399999999999991e-07 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377554+0000", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000142235 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377591+0000", 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6541000000000001e-05 2024-09-17T16:01:57.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377599+0000", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6210000000000004e-06 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377657+0000", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7976999999999997e-05 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.377663+0000", 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6869999999999997e-06 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.378772+0000", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001108946 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.378808+0000", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6288000000000002e-05 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:01:57.506 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0310290279999998, 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047653560000000001, 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:01:57.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393131+0000", 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9600000000000008e-07 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394022+0000", 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089089400000000004 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394066+0000", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4193000000000002e-05 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394089+0000", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2608000000000001e-05 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394165+0000", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5638999999999999e-05 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394175+0000", 2024-09-17T16:01:57.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0277e-05 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397846+0000", 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036707889999999998 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397896+0000", 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9960000000000003e-05 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.0275202410000004, 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020037330000000002, 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.29e-07 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396866+0000", 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022683499999999999 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396945+0000", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8825999999999994e-05 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396956+0000", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1301000000000001e-05 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397067+0000", 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110486 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397078+0000", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1501999999999999e-05 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398596+0000", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015176580000000001 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398643+0000", 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6896e-05 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.515 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.515 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.515 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:57.435+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:57.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:01:57.517 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:57.514+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:57.617 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:57.615+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:57.665 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:57.665 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:57.665 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:01:57.717 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:01:57.715+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:01:57.818 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:57.816+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:57.821 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:57.821 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:57.821 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:57.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:01:57.919 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:57.918+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:57.978 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:7, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.122188515000001, 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001460104, 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.979 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853944+0000", 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853946+0000", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7230000000000001e-06 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854002+0000", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5677999999999998e-05 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854062+0000", 2024-09-17T16:01:57.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0229999999999998e-05 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854075+0000", 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2747e-05 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854189+0000", 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114376 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854203+0000", 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4100000000000001e-05 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855346+0000", 2024-09-17T16:01:57.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011423010000000001 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855405+0000", 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8949000000000001e-05 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.119805877000001, 2024-09-17T16:01:57.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001864189, 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856328+0000", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3570000000000001e-06 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856466+0000", 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137415 2024-09-17T16:01:57.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856516+0000", 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0541999999999998e-05 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856527+0000", 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:01:57.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4474999999999994e-05 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856611+0000", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0720000000000006e-06 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858143+0000", 2024-09-17T16:01:57.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015325499999999999 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858191+0000", 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000003e-05 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.989 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.119184439000001, 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.993 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.116221947, 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:01:57.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:01:57.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:57.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:57.998 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.112646823999999, 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001910307, 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:57.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863487+0000", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863535+0000", 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7713999999999998e-05 2024-09-17T16:01:58.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863653+0000", 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117757 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863662+0000", 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9779999999999994e-06 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863800+0000", 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013886599999999999 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863813+0000", 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.295e-05 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865357+0000", 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001543464 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865396+0000", 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9629000000000003e-05 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.112591617, 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.003 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.007 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=2, reqid=client.4560.0:61, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.091504792999999, 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018138760000000001, 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9e-08 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884974+0000", 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034541900000000002 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885013+0000", 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000002e-05 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885023+0000", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885103+0000", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9871999999999997e-05 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885112+0000", 2024-09-17T16:01:58.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4769999999999992e-06 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886400+0000", 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012878869999999999 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886442+0000", 2024-09-17T16:01:58.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2202e-05 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=11, reqid=client.4560.0:91, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.068420239999998, 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001672984, 2024-09-17T16:01:58.013 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907713+0000", 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907717+0000", 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6290000000000001e-06 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907950+0000", 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023220000000000001 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907988+0000", 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8900999999999999e-05 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907997+0000", 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9109999999999999e-06 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908086+0000", 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8560000000000006e-05 2024-09-17T16:01:58.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.908108+0000", 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2447e-05 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909346+0000", 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001237782 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.909386+0000", 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9554000000000001e-05 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=17, reqid=client.4560.0:124, at_version=35'5, trim_to=0'0, roll_forward_to=35'3))", 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.027513912, 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016306890000000001, 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.018 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948619+0000", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.948620+0000", 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.238e-06 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949131+0000", 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051077299999999996 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949166+0000", 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4619999999999997e-05 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949174+0000", 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2400000000000007e-06 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949224+0000", 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0534000000000001e-05 2024-09-17T16:01:58.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.949231+0000", 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.229e-06 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950221+0000", 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098999599999999993 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.950250+0000", 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9059000000000001e-05 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.088430792, 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:01:58.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:01:58.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:01:58.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.083878832, 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:01:58.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:01:58.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.049475893, 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019463340000000001, 2024-09-17T16:01:58.033 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9200000000000002e-07 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037e-05 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926741+0000", 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0134999999999999e-05 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926751+0000", 2024-09-17T16:01:58.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926822+0000", 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0667000000000006e-05 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926854+0000", 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928578+0000", 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017237979999999999 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928603+0000", 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5769e-05 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.062595421999999, 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013950029999999999, 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913538+0000", 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8500000000000002e-07 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913567+0000", 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9609000000000001e-05 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913630+0000", 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2220999999999999e-05 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913646+0000", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6067999999999999e-05 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913738+0000", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2097999999999998e-05 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.913754+0000", 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5668999999999999e-05 2024-09-17T16:01:58.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.914893+0000", 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011396189999999999 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.914933+0000", 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9434000000000002e-05 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.057700123, 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020717940000000001, 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:58.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6899999999999999e-07 2024-09-17T16:01:58.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918457+0000", 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918505+0000", 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7648000000000002e-05 2024-09-17T16:01:58.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8609999999999993e-06 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918721+0000", 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020663600000000001 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918732+0000", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1058e-05 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920477+0000", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017447840000000001 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920505+0000", 2024-09-17T16:01:58.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7815000000000001e-05 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.05740829, 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017398680000000001, 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918725+0000", 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4199999999999995e-07 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918753+0000", 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7325e-05 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918790+0000", 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7764e-05 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918799+0000", 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6459999999999999e-06 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918873+0000", 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4312000000000003e-05 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918883+0000", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6369999999999994e-06 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920423+0000", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015398930000000001 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920465+0000", 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1649000000000001e-05 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.056555873000001, 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001307234, 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919577+0000", 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919578+0000", 2024-09-17T16:01:58.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2200000000000003e-07 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919655+0000", 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7375999999999999e-05 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919705+0000", 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9598999999999997e-05 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919716+0000", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1274000000000001e-05 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919782+0000", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5438999999999995e-05 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.919791+0000", 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.482e-06 2024-09-17T16:01:58.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920845+0000", 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010542730000000001 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920884+0000", 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8968999999999999e-05 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.051864220000001, 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020194549999999999, 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1600000000000002e-07 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924293+0000", 2024-09-17T16:01:58.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924390+0000", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6893999999999998e-05 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924399+0000", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924465+0000", 2024-09-17T16:01:58.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309000000000005e-05 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924474+0000", 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5289999999999997e-06 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926235+0000", 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001760529 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926288+0000", 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3263000000000001e-05 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.051086471, 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016959259999999999, 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925047+0000", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8700000000000004e-07 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925071+0000", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3918999999999999e-05 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925131+0000", 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.012e-05 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925142+0000", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0481e-05 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925202+0000", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9728999999999997e-05 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925211+0000", 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926703+0000", 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014917629999999999 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926742+0000", 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9305999999999999e-05 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s6 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:58.068 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6031317659999997, 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003094585, 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:01:58.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.61e-07 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373230+0000", 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022871100000000001 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373269+0000", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8470999999999998e-05 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373278+0000", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8449999999999998e-06 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373359+0000", 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0401999999999994e-05 2024-09-17T16:01:58.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373368+0000", 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1919999999999994e-06 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376052+0000", 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026842319999999999 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376096+0000", 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3571e-05 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5793067819999997, 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020885859999999999, 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397062+0000", 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236006 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397106+0000", 2024-09-17T16:01:58.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3995000000000001e-05 2024-09-17T16:01:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397119+0000", 2024-09-17T16:01:58.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397201+0000", 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2498999999999997e-05 2024-09-17T16:01:58.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397214+0000", 2024-09-17T16:01:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2819999999999999e-05 2024-09-17T16:01:58.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398870+0000", 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016554829999999999 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398915+0000", 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5083000000000002e-05 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.084 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:58.086 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:01:58.084+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:01:58.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:01:58.187 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:58.185+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:58.196 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:58.196 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:58.196 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:01:58.196 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:58.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:01:58.289 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:58.287+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:58.335 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:155: done (0 left) 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:156: read oid 16 snap 10 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:156: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:157: read oid 22 snap -1 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:157: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 5 current snap is 18 2024-09-17T16:01:58.342 INFO:tasks.rados.rados.0.smithi181.stdout:158: seq_num 80 ranges {81920=589824,671744=573440,1245184=196608} 2024-09-17T16:01:58.343 INFO:tasks.rados.rados.0.smithi181.stdout:158: writing smithi18136759-5 from 81920 to 671744 tid 1 2024-09-17T16:01:58.347 INFO:tasks.rados.rados.0.smithi181.stdout:158: writing smithi18136759-5 from 671744 to 1245184 tid 2 2024-09-17T16:01:58.348 INFO:tasks.rados.rados.0.smithi181.stdout:158: writing smithi18136759-5 from 1245184 to 1441792 tid 3 2024-09-17T16:01:58.348 INFO:tasks.rados.rados.0.smithi181.stdout:159: snap_create 2024-09-17T16:01:58.348 INFO:tasks.rados.rados.0.smithi181.stdout:157: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-17T16:01:58.349 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:58.353 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:58.357 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:58.357 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:01:58.357 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:01:58.357 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:01:58.357 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:58.358 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:58.362 INFO:tasks.rados.rados.0.smithi181.stdout:158: finishing write tid 1 to smithi18136759-5 2024-09-17T16:01:58.362 INFO:tasks.rados.rados.0.smithi181.stdout:158: finishing write tid 2 to smithi18136759-5 2024-09-17T16:01:58.362 INFO:tasks.rados.rados.0.smithi181.stdout:158: finishing write tid 3 to smithi18136759-5 2024-09-17T16:01:58.362 INFO:tasks.rados.rados.0.smithi181.stdout:158: finishing write tid 4 to smithi18136759-5 2024-09-17T16:01:58.364 INFO:tasks.rados.rados.0.smithi181.stdout:158: finishing write tid 5 to smithi18136759-5 2024-09-17T16:01:58.364 INFO:tasks.rados.rados.0.smithi181.stdout:158: oid 5 updating version 0 to 28 2024-09-17T16:01:58.365 INFO:tasks.rados.rados.0.smithi181.stdout:158: oid 5 updating version 28 to 30 2024-09-17T16:01:58.365 INFO:tasks.rados.rados.0.smithi181.stdout:158: oid 5 version 30 is already newer than 29 2024-09-17T16:01:58.365 INFO:tasks.rados.rados.0.smithi181.stdout:158: oid 5 updating version 30 to 31 2024-09-17T16:01:58.365 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 31 (ObjNum 80 snap 18 seq_num 80) dirty exists 2024-09-17T16:01:58.365 INFO:tasks.rados.rados.0.smithi181.stdout:158: left oid 5 (ObjNum 80 snap 18 seq_num 80) 2024-09-17T16:01:58.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:01:58.389 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:58.387+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:58.488 INFO:tasks.rados.rados.0.smithi181.stdout:156: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:01:58.489 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 187155 2024-09-17T16:01:58.489 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:01:58.487+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:01:58.490 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:58.493 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:01:58.519 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:5 3.bs0 3:d4004f6b:::smithi18136759-2:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.664599633999998, 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010865848000000001, 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.520 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 5 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852345+0000", 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2099999999999997e-07 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852349+0000", 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3249999999999999e-06 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852356+0000", 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6040000000000004e-06 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852361+0000", 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.437e-06 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853722+0000", 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00136107 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854082+0000", 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00036069599999999998 2024-09-17T16:01:58.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854755+0000", 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00067232100000000003 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856366+0000", 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016111039999999999 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863131+0000", 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067652729999999996 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863211+0000", 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9497000000000001e-05 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.664577409, 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.527 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:01:58.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:19 3.1s0 3:8347559a:::smithi18136759-7:head [append 0~114688,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.664535019999999, 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010852832999999999, 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 19 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852409+0000", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852410+0000", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852655+0000", 2024-09-17T16:01:58.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024543600000000003 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852672+0000", 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6328e-05 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852682+0000", 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0365e-05 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852763+0000", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0810999999999996e-05 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852923+0000", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016075700000000001 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.853852+0000", 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00092845700000000005 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856526+0000", 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026744960000000002 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863195+0000", 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066689100000000001 2024-09-17T16:01:58.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863262+0000", 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6847000000000006e-05 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:58.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.66422845, 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:58.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:01:58.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:01:58.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:01:58.545 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.664188319000001, 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:01:58.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:01:58.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.651511025000001, 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:01:58.551 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:01:58.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.557 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.651237783999999, 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:01:58.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:01:58.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.562 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.648372001999999, 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:01:58.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:01:58.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.648320075000001, 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:01:58.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:01:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:01:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:66 3.6s0 3:60a7d8b5:::smithi18136759-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.636380078999998, 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010929104, 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.574 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 66 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880564+0000", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880565+0000", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880568+0000", 2024-09-17T16:01:58.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1939999999999998e-06 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880578+0000", 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7999999999999993e-06 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.880598+0000", 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9964999999999999e-05 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882449+0000", 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018505730000000001 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.882543+0000", 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4135999999999995e-05 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890649+0000", 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081062559999999992 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.890748+0000", 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9234000000000006e-05 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891493+0000", 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074507600000000001 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.579 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.627779309000001, 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:01:58.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:01:58.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.627696135000001, 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:01:58.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:01:58.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.613253367999999, 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.591 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.613213678000001, 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.597 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:99 3.7s0 3:ec861466:::smithi18136759-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.603351976999999, 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011511970999999999, 2024-09-17T16:01:58.602 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 99 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913592+0000", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913593+0000", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913596+0000", 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2140000000000001e-06 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913605+0000", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1060000000000006e-06 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.913612+0000", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2790000000000004e-06 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.915969+0000", 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023571690000000001 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.916049+0000", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9777000000000005e-05 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924068+0000", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0080191269999999992 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.924152+0000", 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4005999999999995e-05 2024-09-17T16:01:58.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.925104+0000", 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095257899999999995 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:122 3.bs0 3:dee30a9d:::smithi18136759-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.573781907000001, 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011179774, 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.607 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 122 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943162+0000", 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943163+0000", 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2200000000000004e-07 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943166+0000", 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3620000000000002e-06 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943176+0000", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.166e-06 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943181+0000", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.502e-06 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944736+0000", 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015548140000000001 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.944814+0000", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7945e-05 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945207+0000", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039347299999999998 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945618+0000", 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00041055900000000001 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954241+0000", 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.008622691 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954342+0000", 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010164 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:123 3.bs0 3:dee30a9d:::smithi18136759-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:01:58.612 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.573753591999999, 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012212639000000001, 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 123 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.613 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943191+0000", 2024-09-17T16:01:58.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943194+0000", 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.565e-06 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943202+0000", 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0050000000000004e-06 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.943207+0000", 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3260000000000003e-06 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945420+0000", 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022128310000000002 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.945469+0000", 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8897999999999998e-05 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954714+0000", 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092452019999999992 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.954800+0000", 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5541e-05 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.955403+0000", 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060376199999999998 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:01:58.618 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.577971438000001, 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:388 3.16s0 3:6d55c691:::smithi18136759-16:c [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:01:58.622 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.17880059000000001, 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.148899007, 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 388 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338147+0000", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.129e-06 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338155+0000", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1389999999999998e-06 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338160+0000", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338288+0000", 2024-09-17T16:01:58.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128374 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338295+0000", 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7730000000000002e-06 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.342955+0000", 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046598580000000002 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:01:58.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343006+0000", 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1329000000000003e-05 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487043+0000", 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14403680199999999 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.16s0 60/53 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-17T16:01:58.627 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.173369881, 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14348618299999999, 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:01:58.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343575+0000", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6400000000000003e-07 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343629+0000", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4472000000000003e-05 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486547+0000", 2024-09-17T16:01:58.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.142917403 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486554+0000", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.503e-06 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486609+0000", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4527000000000002e-05 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487061+0000", 2024-09-17T16:01:58.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045181399999999998 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:01:58.631 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:01:58.633 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:01:58.633 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:58.631+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:58.733 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:58.731+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:58.743 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:58.743 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:58.744 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:01:58.744 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:58.753 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:01:58.834 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:58.832+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:58.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:01:58.857 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:58.857 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:01:58.857 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:01:58.857 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:01:58.857 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:58.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:01:58.935 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:58.933+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:58.973 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:01:58.973 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:01:58.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:41 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'19)", 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.046487034999998, 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077219050000000003, 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923885+0000", 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:58.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923947+0000", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1790000000000003e-05 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923974+0000", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7368999999999998e-05 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923980+0000", 2024-09-17T16:01:58.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924022+0000", 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924037+0000", 2024-09-17T16:01:58.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4127e-05 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931579+0000", 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075422340000000001 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931606+0000", 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7307000000000001e-05 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:58.978 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:42 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'24, mlcod=25'19)", 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.045964204999997, 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072286889999999999, 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:58.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924407+0000", 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924408+0000", 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0699999999999997e-07 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:58.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924412+0000", 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924432+0000", 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9870000000000001e-05 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924436+0000", 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6e-06 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924471+0000", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4527999999999999e-05 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924487+0000", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6303999999999998e-05 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931609+0000", 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071217429999999998 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931636+0000", 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6911e-05 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.063148511000001, 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.983 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:01:58.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:01:58.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.0616786, 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:01:58.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:01:58.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:01:58.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:58.991 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.055325885, 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:58.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:01:58.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:58.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:01:58.997 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.053343298, 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:58.998 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:58.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:01:59.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.052917430000001, 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.004 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.052889260000001, 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.010 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:01:59.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:01:59.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.4s1 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.041726105, 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077955969999999996, 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928646+0000", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928660+0000", 2024-09-17T16:01:59.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4535000000000001e-05 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928684+0000", 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3938e-05 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928692+0000", 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928739+0000", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7574999999999998e-05 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928746+0000", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936413+0000", 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076669930000000004 2024-09-17T16:01:59.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936441+0000", 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7413999999999999e-05 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.040088407000001, 2024-09-17T16:01:59.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:59.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:01:59.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.024 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.039768779999999, 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:59.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:01:59.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.039676966, 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:01:59.029 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:01:59.031 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.033 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.050930016000001, 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0072413809999999999, 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:59.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919441+0000", 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919442+0000", 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8500000000000001e-07 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919460+0000", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7685000000000001e-05 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919508+0000", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8532999999999998e-05 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.919519+0000", 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0883000000000001e-05 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920205+0000", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068607399999999995 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920213+0000", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926416+0000", 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062024070000000001 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.926683+0000", 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00026676199999999998 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=30, last_complete=46'18, committed=1, applied=1))", 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.050283664, 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24942447400000001, 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.038 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920091+0000", 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6620000000000001e-06 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168910+0000", 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24881899499999999 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168924+0000", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3968000000000001e-05 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168997+0000", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3919999999999997e-05 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.169512+0000", 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051455700000000001 2024-09-17T16:01:59.041 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.049071817, 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080140109999999997, 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921300+0000", 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921319+0000", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8519e-05 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2544999999999998e-05 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921346+0000", 2024-09-17T16:01:59.044 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9420000000000003e-06 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921389+0000", 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2620000000000002e-05 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921394+0000", 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3519999999999997e-06 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.045 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929283+0000", 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007889007 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929313+0000", 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0460999999999999e-05 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:59.046 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.045366724000001, 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010009911, 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:59.047 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925020+0000", 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.048 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925042+0000", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2407000000000001e-05 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925050+0000", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3089999999999998e-06 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925097+0000", 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7763999999999999e-05 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.049 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925103+0000", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0759999999999999e-06 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098665709999999993 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935014+0000", 2024-09-17T16:01:59.050 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4459000000000002e-05 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 46/32 ECSubWrite(tid=186, reqid=osd.11.0:184, at_version=0'0, trim_to=0'0, roll_forward_to=46'22))", 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.043196384, 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077139770000000003, 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.051 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927175+0000", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927176+0000", 2024-09-17T16:01:59.052 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9400000000000004e-07 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927186+0000", 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9259999999999995e-06 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927212+0000", 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6821999999999998e-05 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.053 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927220+0000", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8390000000000007e-06 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927269+0000", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9125999999999997e-05 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927278+0000", 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6819999999999992e-06 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.054 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934846+0000", 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075681070000000001 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934889+0000", 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2681e-05 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.055 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=49, reqid=osd.14.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.043153457000001, 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077062470000000003, 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.056 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927218+0000", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5499999999999998e-07 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927227+0000", 2024-09-17T16:01:59.057 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6940000000000001e-06 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927241+0000", 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4260999999999999e-05 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927246+0000", 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6689999999999997e-06 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.058 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927292+0000", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6400999999999998e-05 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927301+0000", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0240000000000003e-06 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934896+0000", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075949219999999996 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.059 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934924+0000", 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7620999999999999e-05 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.040596603999999, 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078830700000000007, 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.060 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.061 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929783+0000", 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3150000000000003e-06 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929802+0000", 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9959999999999999e-05 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.062 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929810+0000", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3069999999999997e-06 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929880+0000", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0123000000000006e-05 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929890+0000", 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.063 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937625+0000", 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077345599999999997 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937658+0000", 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2662e-05 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.064 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 46/32 ECSubWrite(tid=50, reqid=osd.14.0:50, at_version=46'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 14.040308075, 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076204879999999999, 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.065 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930063+0000", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930064+0000", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930082+0000", 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8311000000000001e-05 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.066 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930099+0000", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6895999999999999e-05 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930108+0000", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7870000000000007e-06 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930158+0000", 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9783000000000001e-05 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.067 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.930166+0000", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6889999999999993e-06 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937661+0000", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074941870000000002 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937684+0000", 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3108999999999998e-05 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.068 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.069 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.069 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:01:59.069 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:01:59.069 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:01:59.070 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:01:59.070 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:01:59.071 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:01:59.071 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:01:59.071 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:01:59.072 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:01:59.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:01:59.072 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:01:59.073 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:01:59.073 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:59.068+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:01:59.075 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:01:59.075 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:01:59.097 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:01:59.097 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:01:59.119 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:01:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:01:59.140 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:01:59.141 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:01:59.161 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:01:59.161 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:01:59.170 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:59.169+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:59.182 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:01:59.183 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:01:59.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:01:59.205 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:01:59.215 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.215 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:01:59.226 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:01:59.226 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:01:59.247 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:01:59.247 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:01:59.268 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:01:59.268 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:01:59.271 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:01:59.270+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:01:59.274 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:01:59.272+0000 7f740bfff640 0 -- 172.21.15.6:0/699211591 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7f740c2213a0 msgr2=0x7f740c223790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:01:59.294 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:01:59.295 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:01:59.316 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:01:59.317 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:01:59.328 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.328 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:156: done (3 left) 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:157: done (2 left) 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:158: done (1 left) 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:159: done (0 left) 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:160: read oid 50 snap 7 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:160: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T16:01:59.341 INFO:tasks.rados.rados.0.smithi181.stdout:161: delete oid 14 current snap is 19 2024-09-17T16:01:59.347 INFO:tasks.rados.rados.0.smithi181.stdout:161: done (1 left) 2024-09-17T16:01:59.347 INFO:tasks.rados.rados.0.smithi181.stdout:162: rmattr oid 42 current snap is 19 2024-09-17T16:01:59.347 INFO:tasks.rados.rados.0.smithi181.stdout:162: done (1 left) 2024-09-17T16:01:59.347 INFO:tasks.rados.rados.0.smithi181.stdout:163: snap_remove snap 3 2024-09-17T16:01:59.372 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:01:59.370+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:01:59.443 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.444 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.452 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:01:59.473 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:59.471+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:59.556 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.556 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:01:59.573 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:01:59.572+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:01:59.667 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.667 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.675 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:01:59.673+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:01:59.676 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:01:59.777 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:01:59.775+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:01:59.819 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.819 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.828 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:01:59.877 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:01:59.876+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:01:59.931 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:01:59.931 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:01:59.940 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:01:59.978 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:01:59.977+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:00.042 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.042 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.051 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:00.079 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:02:00.078+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:02:00.155 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.155 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:00.179 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:00.178+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:00.278 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.278 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.281 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:00.279+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:00.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:00.343 INFO:tasks.rados.rados.0.smithi181.stdout:163: done (1 left) 2024-09-17T16:02:00.343 INFO:tasks.rados.rados.0.smithi181.stdout:164: copy_from oid 35 from oid 49 current snap is 19 2024-09-17T16:02:00.343 INFO:tasks.rados.rados.0.smithi181.stdout:160: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T16:02:00.343 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:00.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:00.356 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:00.370 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:00.381 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:00.379+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:00.383 INFO:tasks.rados.rados.0.smithi181.stdout:164: finishing copy_from racing read to smithi18136759-35 2024-09-17T16:02:00.383 INFO:tasks.rados.rados.0.smithi181.stdout:160: done (1 left) 2024-09-17T16:02:00.383 INFO:tasks.rados.rados.0.smithi181.stdout:165: setattr oid 38 current snap is 19 2024-09-17T16:02:00.385 INFO:tasks.rados.rados.0.smithi181.stdout:166: read oid 19 snap -1 2024-09-17T16:02:00.385 INFO:tasks.rados.rados.0.smithi181.stdout:166: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-17T16:02:00.385 INFO:tasks.rados.rados.0.smithi181.stdout:167: read oid 33 snap -1 2024-09-17T16:02:00.385 INFO:tasks.rados.rados.0.smithi181.stdout:167: expect (ObjNum 79 snap 17 seq_num 79) 2024-09-17T16:02:00.386 INFO:tasks.rados.rados.0.smithi181.stdout:168: delete oid 9 current snap is 19 2024-09-17T16:02:00.391 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 21 (ObjNum 1988074720 snap 32637 seq_num 304) dirty exists 2024-09-17T16:02:00.397 INFO:tasks.rados.rados.0.smithi181.stdout:165: done (4 left) 2024-09-17T16:02:00.397 INFO:tasks.rados.rados.0.smithi181.stdout:168: done (3 left) 2024-09-17T16:02:00.397 INFO:tasks.rados.rados.0.smithi181.stdout:169: snap_create 2024-09-17T16:02:00.413 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.413 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:00.506 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:00.505+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:00.566 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.567 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:02:00.616 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:02:00.615+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:02:00.683 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:00.682+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:00.719 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.719 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-17T16:02:00.784 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:00.783+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:00.831 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.831 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-17T16:02:00.885 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:00.883+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:00.904 INFO:tasks.rados.rados.0.smithi181.stdout:167: expect (ObjNum 79 snap 17 seq_num 79) 2024-09-17T16:02:00.904 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:00.925 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:00.955 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1817298 2024-09-17T16:02:00.984 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:00.984 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:00.985 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:00.983+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:00.992 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:01.028 INFO:tasks.rados.rados.0.smithi181.stdout:164: finishing copy_from to smithi18136759-35 2024-09-17T16:02:01.028 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T16:02:01.086 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:02:01.084+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:02:01.131 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:01.131 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:01.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:01.160 INFO:tasks.rados.rados.0.smithi181.stdout:166: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-17T16:02:01.160 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.163 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.166 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.187 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:02:01.185+0000 7fd1aa33b640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33266) UID: 0 2024-09-17T16:02:01.262 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:01.262 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:01.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:01.288 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:01.286+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:01.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:01.389 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:01.387+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:01.392 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:01.396 INFO:tasks.rados.rados.0.smithi181.stdout:164: done (3 left) 2024-09-17T16:02:01.396 INFO:tasks.rados.rados.0.smithi181.stdout:166: done (2 left) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:167: done (1 left) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:169: done (0 left) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:170: read oid 50 snap -1 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:170: expect (ObjNum 71 snap 13 seq_num 71) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:171: read oid 13 snap -1 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:171: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:172: read oid 30 snap -1 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:172: expect (ObjNum 78 snap 16 seq_num 78) 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 4 current snap is 20 2024-09-17T16:02:01.397 INFO:tasks.rados.rados.0.smithi181.stdout:173: seq_num 83 ranges {0=638976,638976=622592,1261568=147456} 2024-09-17T16:02:01.400 INFO:tasks.rados.rados.0.smithi181.stdout:173: writing smithi18136759-4 from 0 to 638976 tid 1 2024-09-17T16:02:01.403 INFO:tasks.rados.rados.0.smithi181.stdout:173: writing smithi18136759-4 from 638976 to 1261568 tid 2 2024-09-17T16:02:01.405 INFO:tasks.rados.rados.0.smithi181.stdout:173: writing smithi18136759-4 from 1261568 to 1409024 tid 3 2024-09-17T16:02:01.405 INFO:tasks.rados.rados.0.smithi181.stdout:174: read oid 20 snap -1 2024-09-17T16:02:01.405 INFO:tasks.rados.rados.0.smithi181.stdout:174: expect (ObjNum 61 snap 8 seq_num 61) 2024-09-17T16:02:01.405 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 40 current snap is 20 2024-09-17T16:02:01.405 INFO:tasks.rados.rados.0.smithi181.stdout:175: seq_num 84 ranges {1556480=655360,2211840=638976,2850816=131072} 2024-09-17T16:02:01.418 INFO:tasks.rados.rados.0.smithi181.stdout:175: writing smithi18136759-40 from 1556480 to 2211840 tid 1 2024-09-17T16:02:01.682 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:01.682 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:01.683 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:01.683 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:01.683 INFO:tasks.rados.rados.0.smithi181.stdout:175: writing smithi18136759-40 from 2211840 to 2850816 tid 2 2024-09-17T16:02:01.683 INFO:tasks.rados.rados.0.smithi181.stdout:175: writing smithi18136759-40 from 2850816 to 2981888 tid 3 2024-09-17T16:02:01.684 INFO:tasks.rados.rados.0.smithi181.stdout:176: copy_from oid 37 from oid 31 current snap is 20 2024-09-17T16:02:01.684 INFO:tasks.rados.rados.0.smithi181.stdout:171: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-17T16:02:01.684 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 89792 2024-09-17T16:02:01.684 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.684 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:01.684 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:01.685 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:01.685 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:01.685 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 89478 2024-09-17T16:02:01.685 INFO:tasks.rados.rados.0.smithi181.stdout:172: expect (ObjNum 78 snap 16 seq_num 78) 2024-09-17T16:02:01.685 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.686 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:01.686 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:01.687 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:01.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:01.688 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:01.688 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:01.689 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:01.689 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:01.690 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.690 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 2113480 2024-09-17T16:02:01.690 INFO:tasks.rados.rados.0.smithi181.stdout:173: finishing write tid 1 to smithi18136759-4 2024-09-17T16:02:01.690 INFO:tasks.rados.rados.0.smithi181.stdout:173: finishing write tid 2 to smithi18136759-4 2024-09-17T16:02:01.690 INFO:tasks.rados.rados.0.smithi181.stdout:173: finishing write tid 3 to smithi18136759-4 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: finishing write tid 4 to smithi18136759-4 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:174: expect (ObjNum 61 snap 8 seq_num 61) 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 642298 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 774139 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: finishing write tid 5 to smithi18136759-4 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: oid 4 updating version 0 to 33 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: oid 4 updating version 33 to 35 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: oid 4 version 35 is already newer than 34 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: oid 4 updating version 35 to 36 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 36 (ObjNum 83 snap 20 seq_num 83) dirty exists 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:173: left oid 4 (ObjNum 83 snap 20 seq_num 83) 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:170: expect (ObjNum 71 snap 13 seq_num 71) 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 3102217 2024-09-17T16:02:01.691 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:01.692 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 490969 2024-09-17T16:02:01.692 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-09-17T16:02:01.692 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:01.680+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:01.692 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-17T16:02:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:01.693 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:01.693 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:01.694 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:01.694 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:01.711 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:01.712 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:175: finishing write tid 1 to smithi18136759-40 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:175: finishing write tid 2 to smithi18136759-40 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:175: finishing write tid 3 to smithi18136759-40 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:176: finishing copy_from racing read to smithi18136759-37 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:176: finishing copy_from to smithi18136759-37 2024-09-17T16:02:01.716 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 37 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:170: done (6 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:171: done (5 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:172: done (4 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:173: done (3 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:174: done (2 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:176: done (1 left) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:177: read oid 37 snap -1 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:177: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-17T16:02:01.717 INFO:tasks.rados.rados.0.smithi181.stdout:178: snap_remove snap 13 2024-09-17T16:02:01.732 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:01.753 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:01.754 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:01.775 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:01.775 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:01.775 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:01.773+0000 7efd837fe640 0 -- 172.21.15.6:0/1295896969 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7efd842248a0 msgr2=0x7efd84226c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:01.782 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:01.781+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:01.821 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:01.821 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:01.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:01.884 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:01.882+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:01.975 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:01.975 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:01.983 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:01.985 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:01.983+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:01.997 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:01.997 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:02.085 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:02.083+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:02.088 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.225857089000002, 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053801109999999999, 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.089 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:02.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860418+0000", 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020234499999999999 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860448+0000", 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0692999999999999e-05 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860458+0000", 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6460000000000005e-06 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860528+0000", 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0272999999999996e-05 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860536+0000", 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1040000000000007e-06 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865550+0000", 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050133510000000001 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865595+0000", 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5396999999999999e-05 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.224695448999999, 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042468009999999997, 2024-09-17T16:02:02.094 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861377+0000", 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861378+0000", 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8199999999999999e-07 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861403+0000", 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5069999999999999e-05 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861434+0000", 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1396000000000002e-05 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861442+0000", 2024-09-17T16:02:02.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4500000000000004e-06 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861497+0000", 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4997000000000002e-05 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861508+0000", 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0360000000000001e-05 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865601+0000", 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040936439999999996 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865624+0000", 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2101999999999999e-05 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:02.099 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.221821365, 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051579750000000004, 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864260+0000", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5769999999999994e-06 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864296+0000", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6227999999999999e-05 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864304+0000", 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0090000000000007e-06 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864392+0000", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8549999999999998e-05 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864402+0000", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3279999999999995e-06 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869377+0000", 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004975045 2024-09-17T16:02:02.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869409+0000", 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1953999999999999e-05 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:02.104 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.21860895, 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:02.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:02:02.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:02:02.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:02:02.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:02.109 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.218219238, 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:02:02.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:02:02.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:02.114 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.208798828999999, 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:02.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:02:02.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:02:02.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.154113303999999, 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:02:02.119 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:02:02.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:02:02.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:02.126 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.153730905, 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:02:02.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:02:02.132 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.153695112000001, 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.133 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:02:02.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:02:02.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.153666947000001, 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:02:02.139 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:02:02.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:02:02.144 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.153393428000001, 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.145 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.153382371999999, 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:02.151 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:02:02.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:02:02.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.156 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.120627218999999, 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.157 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:02:02.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.120138169000001, 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.163 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:02:02.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:02:02.170 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.120097318999999, 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.171 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:02:02.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.176 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 39/32 ECSubWrite(tid=157, reqid=osd.11.0:157, at_version=39'20, trim_to=0'0, roll_forward_to=38'15))", 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:37.878008+0000", 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.208063854999999, 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004253929, 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878008+0000", 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878008+0000", 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878009+0000", 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6300000000000004e-07 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878034+0000", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4655e-05 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878079+0000", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5562000000000002e-05 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878089+0000", 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2e-06 2024-09-17T16:02:02.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878174+0000", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5045999999999999e-05 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.878186+0000", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2703e-05 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.882229+0000", 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040428840000000001 2024-09-17T16:02:02.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.882262+0000", 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3015999999999997e-05 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:37.892288+0000", 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.193784081, 2024-09-17T16:02:02.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042122549999999998, 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892288+0000", 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892288+0000", 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892289+0000", 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7100000000000001e-07 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892300+0000", 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1002000000000001e-05 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892329+0000", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9388999999999999e-05 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892339+0000", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9709999999999999e-06 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892396+0000", 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6616000000000001e-05 2024-09-17T16:02:02.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.892405+0000", 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0010000000000007e-06 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.896467+0000", 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040623550000000001 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:37.896500+0000", 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3149999999999999e-05 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:02.186 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.172377723, 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073441670000000004, 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:02.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913695+0000", 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913725+0000", 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9743999999999998e-05 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919758+0000", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00603318 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919766+0000", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3610000000000001e-06 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919835+0000", 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9336000000000002e-05 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919844+0000", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921001+0000", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115653 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921039+0000", 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7700000000000002e-05 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.190 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 46/32 ECSubWriteReply(tid=17, last_complete=46'12, committed=1, applied=1))", 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.162391146000001, 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.330502611, 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:02.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923682+0000", 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923686+0000", 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253962+0000", 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33027621400000001 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253975+0000", 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254119+0000", 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014402100000000001 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254184+0000", 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4158999999999993e-05 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.194 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:346 3.3s0 3:c1daf1ed:::smithi18136759-38:head [delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7273023250000001, 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069854690000000002, 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.195 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358772+0000", 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0810000000000001e-06 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358779+0000", 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2849999999999997e-06 2024-09-17T16:02:02.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358790+0000", 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9759999999999995e-06 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358838+0000", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9730000000000001e-05 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358966+0000", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012749199999999999 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.359434+0000", 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046890000000000001 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.362289+0000", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028544030000000001 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365699+0000", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034100210000000001 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365755+0000", 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6456000000000003e-05 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.201 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.201 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:02.201 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:02.201 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:02.202 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:02.200+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:02.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:02.304 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:02.302+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:02.312 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:02.313 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:02.313 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:02.313 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:02.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:02.338 INFO:tasks.rados.rados.0.smithi181.stdout:177: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-17T16:02:02.339 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 144466 2024-09-17T16:02:02.343 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 247617 2024-09-17T16:02:02.344 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:02.349 INFO:tasks.rados.rados.0.smithi181.stdout:175: finishing write tid 4 to smithi18136759-40 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: finishing write tid 5 to smithi18136759-40 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: oid 40 updating version 0 to 38 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: oid 40 updating version 38 to 39 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: oid 40 updating version 39 to 41 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: oid 40 version 41 is already newer than 40 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 41 (ObjNum 84 snap 20 seq_num 84) dirty exists 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: left oid 40 (ObjNum 84 snap 20 seq_num 84) 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:175: done (2 left) 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:177: done (1 left) 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:178: done (0 left) 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:179: read oid 16 snap -1 2024-09-17T16:02:02.350 INFO:tasks.rados.rados.0.smithi181.stdout:179: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:02:02.351 INFO:tasks.rados.rados.0.smithi181.stdout:180: snap_create 2024-09-17T16:02:02.352 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-17T16:02:02.352 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:02.352 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-17T16:02:02.352 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-17T16:02:02.352 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:02:02.384 INFO:tasks.ceph.osd.7:Stopped 2024-09-17T16:02:02.384 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-17T16:02:02.385 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:02:02.385 DEBUG:teuthology.orchestra.run.smithi181:> type ceph-objectstore-tool 2024-09-17T16:02:02.426 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:02.424+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:02.439 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-17T16:02:02.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-17T16:02:02.442 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:02.442 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:02.442 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:02.442 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:02.442 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:02.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:02.524 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:02.522+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:234 3.fs0 3:fe5cbd82:::smithi18136759-11:head [rollback 5] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.921314+0000", 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.634691773, 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067827470000000004, 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.559 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 234 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921314+0000", 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921314+0000", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.342e-06 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921320+0000", 2024-09-17T16:02:02.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0000000000000004e-06 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921331+0000", 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0926e-05 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921337+0000", 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7020000000000004e-06 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921387+0000", 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9217e-05 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921449+0000", 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1808000000000002e-05 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921939+0000", 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00049095599999999997 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.925246+0000", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033067880000000002 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928056+0000", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002809443 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928096+0000", 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0565000000000002e-05 2024-09-17T16:02:02.564 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:235 3.fs0 3:fe5cbd82:::smithi18136759-11:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.63465377, 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072733650000000004, 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8500000000000002e-07 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921355+0000", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.084e-06 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921360+0000", 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5079999999999999e-06 2024-09-17T16:02:02.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921363+0000", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1109999999999999e-06 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922189+0000", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082667099999999996 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922288+0000", 2024-09-17T16:02:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8857999999999994e-05 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922688+0000", 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00039985699999999999 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.927817+0000", 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005129013 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928567+0000", 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00075011800000000003 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928625+0000", 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7760000000000003e-05 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:248 3.cs0 3:3808462d:::smithi18136759-6:head [copy-from ver 2] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:02.570 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.627423373999999, 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019268496999999999, 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 248 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:02.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928583+0000", 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928588+0000", 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2220000000000001e-06 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928601+0000", 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2717e-05 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928608+0000", 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8600000000000004e-06 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928671+0000", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3791e-05 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928756+0000", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4293000000000001e-05 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940817+0000", 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01206144 2024-09-17T16:02:02.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.942933+0000", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021155280000000002 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947791+0000", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048580639999999996 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947850+0000", 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9682999999999998e-05 2024-09-17T16:02:02.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:249 3.cs0 3:3808462d:::smithi18136759-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.627382262000001, 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013329483, 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.576 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928624+0000", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2699999999999999e-07 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928626+0000", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8940000000000002e-06 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928643+0000", 2024-09-17T16:02:02.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6988999999999999e-05 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928651+0000", 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.267e-06 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928901+0000", 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025058200000000001 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928955+0000", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3973e-05 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941770+0000", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012814828 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941852+0000", 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083000000000007e-05 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941953+0000", 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010034 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:285 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.624199943000001, 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010247877000000001, 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.581 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 285 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931808+0000", 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8779999999999998e-06 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931822+0000", 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3828e-05 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931850+0000", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7500000000000001e-05 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931863+0000", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3738e-05 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931969+0000", 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001052 2024-09-17T16:02:02.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931976+0000", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938082+0000", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061057769999999997 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938176+0000", 2024-09-17T16:02:02.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3714000000000004e-05 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939069+0000", 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00089279400000000003 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941333+0000", 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022647380000000001 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941992+0000", 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065853199999999998 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.942053+0000", 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1447999999999998e-05 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:286 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 2] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:02.587 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.624046252999999, 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013358431, 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 286 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931960+0000", 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:02:02.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931965+0000", 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9010000000000002e-06 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931974+0000", 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2520000000000005e-06 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931981+0000", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5950000000000004e-06 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932071+0000", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0390999999999996e-05 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932079+0000", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2679999999999994e-06 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939566+0000", 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074865030000000003 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939653+0000", 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7336999999999994e-05 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940238+0000", 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058448400000000002 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944090+0000", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00385192 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945232+0000", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001142784 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945317+0000", 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5080999999999994e-05 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.593 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:287 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.624010314, 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013571456000000001, 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 287 2024-09-17T16:02:02.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931996+0000", 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931999+0000", 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4300000000000002e-06 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932006+0000", 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2729999999999997e-06 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932012+0000", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8089999999999998e-06 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932107+0000", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4751000000000001e-05 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932112+0000", 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940481+0000", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083682089999999997 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940546+0000", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5704000000000007e-05 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940923+0000", 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00037686099999999998 2024-09-17T16:02:02.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944112+0000", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031884999999999999 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945503+0000", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001391226 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945566+0000", 2024-09-17T16:02:02.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3510999999999996e-05 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 52/32 ECSubWriteReply(tid=51, last_complete=52'22, committed=1, applied=1))", 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.199034161, 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.138524809, 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356972+0000", 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:02.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356976+0000", 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2950000000000004e-06 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495052+0000", 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13807575 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495065+0000", 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3276e-05 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495133+0000", 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8274000000000004e-05 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495496+0000", 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036245899999999998 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.603 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:344 3.fs0 3:fd9ed431:::smithi18136759-28:head [stat,delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.220042232000001, 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012105121, 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:02.604 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 344 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335966+0000", 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.004e-06 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335978+0000", 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1810999999999999e-05 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336003+0000", 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5344e-05 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336017+0000", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3901e-05 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336125+0000", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107707 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336136+0000", 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1476e-05 2024-09-17T16:02:02.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340410+0000", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004273453 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340525+0000", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115017 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.341450+0000", 2024-09-17T16:02:02.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00092533800000000005 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.345523+0000", 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040730929999999999 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348008+0000", 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024844609999999999 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348068+0000", 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0516000000000001e-05 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.350444+0000", 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.205561162, 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0070386119999999996, 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.610 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350444+0000", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350444+0000", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350444+0000", 2024-09-17T16:02:02.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6400000000000001e-07 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350479+0000", 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4683e-05 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350917+0000", 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043845499999999998 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350925+0000", 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.351062+0000", 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013686500000000001 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.351072+0000", 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0179e-05 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357430+0000", 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063571610000000001 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357483+0000", 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3102000000000001e-05 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.614 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358756+0000", 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.197248923, 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0062433180000000003, 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358756+0000", 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358756+0000", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358757+0000", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7700000000000004e-07 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358769+0000", 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1742999999999999e-05 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358805+0000", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6538999999999997e-05 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358815+0000", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358881+0000", 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6025999999999994e-05 2024-09-17T16:02:02.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358890+0000", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7690000000000002e-06 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.364960+0000", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006070509 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365000+0000", 2024-09-17T16:02:02.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9573000000000002e-05 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=128, reqid=osd.12.0:128, at_version=64'68, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.21810706099999999, 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076656809999999997, 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:02.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337917+0000", 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8366999999999999e-05 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339622+0000", 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017049529999999999 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.621 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339629+0000", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3660000000000003e-06 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339905+0000", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027553099999999997 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339915+0000", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0124e-05 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345540+0000", 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056253780000000003 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799e-05 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:02.623 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.21634857499999999, 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081855320000000006, 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339692+0000", 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4459000000000003e-05 2024-09-17T16:02:02.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339726+0000", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4029000000000002e-05 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339733+0000", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0790000000000003e-06 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340271+0000", 2024-09-17T16:02:02.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00053823 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340282+0000", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347801+0000", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075190999999999999 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347842+0000", 2024-09-17T16:02:02.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1387999999999997e-05 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.10s6 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.21044168299999999, 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011676772, 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:02.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.29e-07 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8580000000000001e-05 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346013+0000", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00042032499999999998 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346021+0000", 2024-09-17T16:02:02.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2269999999999993e-06 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346353+0000", 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033160900000000002 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346363+0000", 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0390999999999999e-05 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357206+0000", 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010843148 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357240+0000", 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4263e-05 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:02.632 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.20953463, 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010663848, 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7099999999999998e-07 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346489+0000", 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7731999999999999e-05 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346664+0000", 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017518199999999999 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346671+0000", 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1239999999999999e-06 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346721+0000", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.02e-05 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.55e-06 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357095+0000", 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010364226000000001 2024-09-17T16:02:02.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357135+0000", 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9563000000000001e-05 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s5 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.208154954, 2024-09-17T16:02:02.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092773109999999999, 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347851+0000", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347861+0000", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0325e-05 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347893+0000", 2024-09-17T16:02:02.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1173000000000001e-05 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347900+0000", 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.306e-06 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347964+0000", 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3678999999999998e-05 2024-09-17T16:02:02.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347973+0000", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0790000000000007e-06 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357093+0000", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091199190000000006 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357128+0000", 2024-09-17T16:02:02.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5055999999999998e-05 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.20687921400000001, 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090524279999999995, 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.642 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349127+0000", 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:02.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349137+0000", 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349186+0000", 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8306999999999997e-05 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349195+0000", 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3640000000000005e-06 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349414+0000", 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021941500000000001 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349424+0000", 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358144+0000", 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087197060000000007 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358179+0000", 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4814999999999998e-05 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=132, reqid=osd.12.0:132, at_version=64'72, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.204424303, 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076285509999999999, 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351582+0000", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351615+0000", 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2962000000000001e-05 2024-09-17T16:02:02.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351654+0000", 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8847999999999998e-05 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351662+0000", 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0099999999999995e-06 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351779+0000", 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117569 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351788+0000", 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5639999999999997e-06 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359157+0000", 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073690580000000004 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359210+0000", 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2769999999999998e-05 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.203558821, 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079374270000000004, 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.651 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352447+0000", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352467+0000", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0146999999999999e-05 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352504+0000", 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6365000000000002e-05 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352512+0000", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352565+0000", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3613999999999999e-05 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352576+0000", 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1141999999999999e-05 2024-09-17T16:02:02.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360356+0000", 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007779584 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360384+0000", 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7847999999999998e-05 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.17s6 64/53 ECSubWrite(tid=138, reqid=osd.13.0:138, at_version=64'29, trim_to=0'0, roll_forward_to=62'27))", 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.353588+0000", 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.202417021, 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067239350000000003, 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353588+0000", 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353588+0000", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353589+0000", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6300000000000004e-07 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353609+0000", 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9796000000000002e-05 2024-09-17T16:02:02.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353642+0000", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2889999999999999e-05 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353658+0000", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5803000000000001e-05 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353729+0000", 2024-09-17T16:02:02.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1575999999999994e-05 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353747+0000", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7890999999999999e-05 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360279+0000", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065315570000000003 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360312+0000", 2024-09-17T16:02:02.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3658999999999997e-05 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:02.660 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:02.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:02.662 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:02.660+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:02.763 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:02.761+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:02.769 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:02.769 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:02.769 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:02.769 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:02.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:02.810 INFO:teuthology.orchestra.run.smithi181.stdout:3.fs4 2024-09-17T16:02:02.810 INFO:teuthology.orchestra.run.smithi181.stdout:3.bs6 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.as2 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.2s6 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.17s2 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.16s1 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.12s6 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.6s1 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.11s5 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.7s2 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.1s6 2024-09-17T16:02:02.811 INFO:teuthology.orchestra.run.smithi181.stdout:3.10s0 2024-09-17T16:02:02.863 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:02.862+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:02.884 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:02.885 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:02.885 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:02.885 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:02.885 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:02.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:02.965 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:02.964+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:03.003 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 39/32 ECSubWrite(tid=13, reqid=osd.15.0:13, at_version=39'10, trim_to=0'0, roll_forward_to=38'8))", 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:37.881175+0000", 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 25.119586387999998, 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004749339, 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881175+0000", 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881175+0000", 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881175+0000", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7499999999999999e-07 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881197+0000", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1698000000000001e-05 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881230+0000", 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3797000000000002e-05 2024-09-17T16:02:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881240+0000", 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1940000000000004e-06 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881301+0000", 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1338000000000001e-05 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.881311+0000", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0567999999999999e-05 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.885871+0000", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045594900000000002 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.885924+0000", 2024-09-17T16:02:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3078999999999998e-05 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:37.887694+0000", 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 25.113067445999999, 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0042394340000000003, 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.887694+0000", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.887694+0000", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.887694+0000", 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6600000000000003e-07 2024-09-17T16:02:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.888317+0000", 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00062279800000000004 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.888351+0000", 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3840000000000001e-05 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.888360+0000", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1840000000000005e-06 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.888437+0000", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6804999999999994e-05 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.888448+0000", 2024-09-17T16:02:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1144999999999999e-05 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.891892+0000", 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034446289999999998 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:37.891933+0000", 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0766999999999998e-05 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.bs5 44/32 ECSubWrite(tid=62, reqid=client.4560.0:258, at_version=44'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.932223+0000", 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 20.068538201999999, 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051520480000000002, 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932223+0000", 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932223+0000", 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932223+0000", 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9800000000000003e-07 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932237+0000", 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3709e-05 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932261+0000", 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4270999999999999e-05 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932267+0000", 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2750000000000003e-06 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932305+0000", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8475e-05 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.932312+0000", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4930000000000003e-06 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.937333+0000", 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050217040000000001 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.937375+0000", 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1523000000000002e-05 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 44/32 ECSubWrite(tid=37, reqid=client.4560.0:236, at_version=44'15, trim_to=0'0, roll_forward_to=44'11))", 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.935861+0000", 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 20.064900518999998, 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041771330000000004, 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.935861+0000", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.935861+0000", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.935861+0000", 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8400000000000004e-07 2024-09-17T16:02:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.936272+0000", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000410407 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.936295+0000", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3328000000000001e-05 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.936302+0000", 2024-09-17T16:02:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.936343+0000", 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1134999999999998e-05 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.936349+0000", 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7290000000000003e-06 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940013+0000", 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036638920000000002 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940038+0000", 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4372e-05 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 44/32 ECSubWrite(tid=38, reqid=client.4560.0:259, at_version=44'13, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.940295+0000", 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 20.06046649, 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0052429929999999996, 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940295+0000", 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940295+0000", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940295+0000", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940656+0000", 2024-09-17T16:02:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036086899999999999 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940691+0000", 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5123000000000002e-05 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940700+0000", 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3289999999999994e-06 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940785+0000", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5405e-05 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940794+0000", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7660000000000004e-06 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.945511+0000", 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047173249999999996 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.945538+0000", 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6434000000000001e-05 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 45/32 ECSubWrite(tid=27, reqid=client.4560.0:266, at_version=45'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:43.922425+0000", 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.078336338, 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003859426, 2024-09-17T16:02:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922425+0000", 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922425+0000", 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922425+0000", 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0500000000000003e-07 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922444+0000", 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8461000000000002e-05 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922475+0000", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1393999999999998e-05 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922484+0000", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0229999999999998e-06 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922544+0000", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0007000000000003e-05 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.922555+0000", 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0934999999999999e-05 2024-09-17T16:02:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.926241+0000", 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036856129999999999 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:43.926284+0000", 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3288000000000003e-05 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=182, reqid=osd.11.0:182, at_version=46'20, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.082901983999999, 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066826129999999996, 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5800000000000001e-07 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917886+0000", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6931999999999999e-05 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918192+0000", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030561400000000001 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918355+0000", 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016328600000000001 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918431+0000", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5690000000000002e-05 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918441+0000", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9450000000000005e-06 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924513+0000", 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060725709999999997 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924542+0000", 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8317e-05 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.082223040999999, 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060611290000000002, 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.96e-07 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0068000000000001e-05 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010144 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918678+0000", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0900000000000005e-06 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918750+0000", 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2100000000000004e-05 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918761+0000", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0767999999999999e-05 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924554+0000", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057936560000000003 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924599+0000", 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4710999999999999e-05 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.082185940999999, 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059181870000000001, 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918576+0000", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918591+0000", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5583999999999999e-05 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918817+0000", 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022566799999999999 2024-09-17T16:02:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918825+0000", 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9389999999999997e-06 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918894+0000", 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9332000000000006e-05 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918904+0000", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9860000000000006e-06 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924448+0000", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055442470000000004 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924493+0000", 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4947000000000002e-05 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.081213331000001, 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056807560000000003, 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919549+0000", 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919571+0000", 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2155999999999999e-05 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919754+0000", 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018332299999999999 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919767+0000", 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2656e-05 2024-09-17T16:02:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920277+0000", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051049799999999996 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920289+0000", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1388e-05 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925184+0000", 2024-09-17T16:02:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048955539999999999 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925228+0000", 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4202000000000001e-05 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.081173546999999, 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056906200000000004, 2024-09-17T16:02:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1299999999999999e-07 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919607+0000", 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9062000000000001e-05 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919796+0000", 2024-09-17T16:02:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018866 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919803+0000", 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4150000000000002e-06 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920325+0000", 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052165800000000004 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920334+0000", 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925242+0000", 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049072009999999999 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925278+0000", 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6634000000000002e-05 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.641999427, 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067241940000000002, 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358763+0000", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1710000000000001e-06 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358778+0000", 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5158e-05 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358817+0000", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8832999999999999e-05 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358826+0000", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2390000000000008e-06 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358909+0000", 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2556e-05 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358920+0000", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0910000000000001e-05 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365448+0000", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006528748 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365486+0000", 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7579000000000001e-05 2024-09-17T16:02:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/53 ECSubWrite(tid=23, reqid=client.4560.0:346, at_version=54'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.641264458, 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059831219999999996, 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359521+0000", 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3705000000000001e-05 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359561+0000", 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9879999999999998e-05 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359570+0000", 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1640000000000007e-06 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359645+0000", 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4765000000000007e-05 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359656+0000", 2024-09-17T16:02:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0702e-05 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365440+0000", 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057839320000000003 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365480+0000", 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0219000000000002e-05 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/53 ECSubWrite(tid=24, reqid=client.4560.0:346, at_version=0'0, trim_to=0'0, roll_forward_to=54'18))", 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.365995+0000", 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.634766318, 2024-09-17T16:02:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004719773, 2024-09-17T16:02:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365995+0000", 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365995+0000", 2024-09-17T16:02:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365996+0000", 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8199999999999999e-07 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.366006+0000", 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0829000000000001e-05 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.366077+0000", 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0801000000000002e-05 2024-09-17T16:02:03.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.366094+0000", 2024-09-17T16:02:03.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6359e-05 2024-09-17T16:02:03.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.366281+0000", 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000187968 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.366293+0000", 2024-09-17T16:02:03.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1361e-05 2024-09-17T16:02:03.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.370680+0000", 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004386839 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.370715+0000", 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4833999999999999e-05 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 60/53 ECSubWrite(tid=200, reqid=client.4560.0:395, at_version=60'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:58.356023+0000", 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.6447378910000001, 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037074199999999999, 2024-09-17T16:02:03.077 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356023+0000", 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356023+0000", 2024-09-17T16:02:03.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356024+0000", 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:03.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356038+0000", 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4613999999999999e-05 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356073+0000", 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4335999999999998e-05 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356081+0000", 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5380000000000002e-06 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356167+0000", 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5421999999999996e-05 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.356178+0000", 2024-09-17T16:02:03.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1287000000000001e-05 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.359682+0000", 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035034609999999998 2024-09-17T16:02:03.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.359731+0000", 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9054999999999997e-05 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.086 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 60/53 ECSubWrite(tid=201, reqid=client.4560.0:395, at_version=0'0, trim_to=0'0, roll_forward_to=60'31))", 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:58.361508+0000", 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.6392532290000004, 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037398000000000002, 2024-09-17T16:02:03.087 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361508+0000", 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361508+0000", 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361509+0000", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4099999999999998e-07 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361517+0000", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5329999999999994e-06 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361549+0000", 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1520999999999999e-05 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361557+0000", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3410000000000003e-06 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361615+0000", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8168999999999998e-05 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.361624+0000", 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9439999999999994e-06 2024-09-17T16:02:03.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.365221+0000", 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035965189999999998 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:58.365248+0000", 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7032000000000001e-05 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.343089+0000", 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.65767189000000004, 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004737911, 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343089+0000", 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343089+0000", 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343089+0000", 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3300000000000001e-07 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343111+0000", 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1954999999999999e-05 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343284+0000", 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000172666 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343292+0000", 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0770000000000008e-06 2024-09-17T16:02:03.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343338+0000", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6252999999999999e-05 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.343365+0000", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6293000000000001e-05 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347790+0000", 2024-09-17T16:02:03.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044254139999999999 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347827+0000", 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7020000000000001e-05 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349285+0000", 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.65147645200000004, 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043063199999999998, 2024-09-17T16:02:03.097 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349285+0000", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349285+0000", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349285+0000", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1199999999999999e-07 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349292+0000", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4410000000000004e-06 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349458+0000", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016555500000000001 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349465+0000", 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.559e-06 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349520+0000", 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4870999999999997e-05 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349529+0000", 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3240000000000003e-06 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353558+0000", 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004029042 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.353591+0000", 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3216000000000002e-05 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:03.101 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352593+0000", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.64816758100000005, 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046907069999999997, 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352593+0000", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352593+0000", 2024-09-17T16:02:03.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352604+0000", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0560999999999999e-05 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352623+0000", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8698999999999998e-05 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352658+0000", 2024-09-17T16:02:03.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5021000000000002e-05 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352666+0000", 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8839999999999994e-06 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352749+0000", 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2966000000000005e-05 2024-09-17T16:02:03.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352763+0000", 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4555e-05 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357250+0000", 2024-09-17T16:02:03.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044863990000000003 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357284+0000", 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4622000000000002e-05 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 64/32 ECSubWrite(tid=134, reqid=osd.12.0:133, at_version=0'0, trim_to=0'0, roll_forward_to=64'74))", 2024-09-17T16:02:03.106 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.360731+0000", 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.64002990999999998, 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045199029999999996, 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360731+0000", 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360731+0000", 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360732+0000", 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9800000000000003e-07 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360751+0000", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8689e-05 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360787+0000", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5865999999999998e-05 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360796+0000", 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1649999999999995e-06 2024-09-17T16:02:03.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360869+0000", 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2908000000000001e-05 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360878+0000", 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7790000000000007e-06 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.365223+0000", 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043444139999999996 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.365251+0000", 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8283999999999999e-05 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.111 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.112 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 3.10s0 --file /var/log/ceph/exp.3.10s0.7 2024-09-17T16:02:03.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:03.115 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:03.111+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:03.214 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:03.212+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:03.221 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.221 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:03.221 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:03.221 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:03.316 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:03.314+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:03.342 INFO:tasks.rados.rados.0.smithi181.stdout:180: done (1 left) 2024-09-17T16:02:03.342 INFO:tasks.rados.rados.0.smithi181.stdout:181: snap_create 2024-09-17T16:02:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:03.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:02:03.417 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:03.414+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:03.488 INFO:teuthology.orchestra.run.smithi181.stderr:Exporting 3.10s0 info 3.10s0( v 64'35 (0'0,64'35] local-lis/les=57/58 n=3 ec=53/32 lis/c=57/57 les/c/f=58/58/0 sis=57) 2024-09-17T16:02:03.489 INFO:teuthology.orchestra.run.smithi181.stderr:Read 0#3:0a7b2816:::smithi18136759-31:9# 2024-09-17T16:02:03.489 INFO:teuthology.orchestra.run.smithi181.stderr:Read 0#3:0a7b2816:::smithi18136759-31:11# 2024-09-17T16:02:03.490 INFO:teuthology.orchestra.run.smithi181.stderr:Read 0#3:0a7b2816:::smithi18136759-31:head# 2024-09-17T16:02:03.490 INFO:teuthology.orchestra.run.smithi181.stderr:Export successful 2024-09-17T16:02:03.501 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.501 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.630771752000001, 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019947230999999999, 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.502 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867810+0000", 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867830+0000", 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9922999999999999e-05 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881759+0000", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01392914 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881768+0000", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7809999999999994e-06 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881836+0000", 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8491999999999994e-05 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881849+0000", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2418e-05 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887709+0000", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058604850000000003 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887756+0000", 2024-09-17T16:02:03.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7168e-05 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=4, reqid=client.4560.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875798+0000", 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.622782975, 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065974859999999996, 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875798+0000", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875798+0000", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875799+0000", 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7899999999999997e-07 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875848+0000", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9376999999999998e-05 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875944+0000", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5953999999999996e-05 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875951+0000", 2024-09-17T16:02:03.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9149999999999999e-06 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876018+0000", 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6890000000000005e-05 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876030+0000", 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2411e-05 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882354+0000", 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063239409999999996 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882395+0000", 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1219e-05 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=5, reqid=client.4560.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.511 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.622727293000001, 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012978303, 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.512 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799999999999999e-07 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875860+0000", 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.02e-06 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875972+0000", 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011225100000000001 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875986+0000", 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3526000000000001e-05 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877064+0000", 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00107855 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877074+0000", 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888801+0000", 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011726927999999999 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888832+0000", 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0750999999999997e-05 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=6, reqid=client.4560.0:52, at_version=35'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.622720086000001, 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012993864000000001, 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.516 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3199999999999999e-07 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875913+0000", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1820999999999997e-05 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876038+0000", 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012561899999999999 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876048+0000", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.166e-06 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877256+0000", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012082309999999999 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877264+0000", 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0399999999999993e-06 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888835+0000", 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011570815 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888855+0000", 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0040000000000001e-05 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=7, reqid=client.4560.0:53, at_version=35'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:03.520 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.622666688999999, 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012966596, 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.86e-07 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875921+0000", 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876450+0000", 2024-09-17T16:02:03.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052973099999999997 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876457+0000", 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1029999999999996e-06 2024-09-17T16:02:03.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879593+0000", 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003135597 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879606+0000", 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3142000000000001e-05 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888861+0000", 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092553359999999994 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888881+0000", 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9330999999999998e-05 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=12, reqid=client.4560.0:64, at_version=35'3, trim_to=0'0, roll_forward_to=35'1))", 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.617089636999999, 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069303020000000002, 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.530 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:03.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881492+0000", 2024-09-17T16:02:03.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1060000000000001e-06 2024-09-17T16:02:03.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881730+0000", 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00023725499999999999 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882445+0000", 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071539399999999995 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882453+0000", 2024-09-17T16:02:03.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2830000000000001e-06 2024-09-17T16:02:03.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882512+0000", 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8569000000000001e-05 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882521+0000", 2024-09-17T16:02:03.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1290000000000002e-06 2024-09-17T16:02:03.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.538 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888395+0000", 2024-09-17T16:02:03.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058743440000000001 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888422+0000", 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6222000000000001e-05 2024-09-17T16:02:03.539 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:61 3.5s0 3:a2653429:::smithi18136759-21:head [append 0~344064,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.540 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.615737784, 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075767569999999999, 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.541 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 61 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:03.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882845+0000", 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5650000000000001e-06 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883323+0000", 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047814299999999999 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883338+0000", 2024-09-17T16:02:03.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.504e-05 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883346+0000", 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0760000000000003e-06 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5992999999999997e-05 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883498+0000", 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6426000000000001e-05 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.884493+0000", 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00099483199999999996 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889296+0000", 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048030460000000001 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890335+0000", 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001038546 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890420+0000", 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.509e-05 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.550 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:62 3.5s0 3:a2653429:::smithi18136759-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.615215949, 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007574038, 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.551 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 62 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883366+0000", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883374+0000", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1920000000000005e-06 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883384+0000", 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0409e-05 2024-09-17T16:02:03.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883390+0000", 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887851+0000", 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044615319999999998 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887944+0000", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2359000000000001e-05 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888411+0000", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00046751900000000001 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889298+0000", 2024-09-17T16:02:03.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088712999999999997 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890854+0000", 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001555753 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890939+0000", 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4860999999999999e-05 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.556 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:63 3.5s0 3:a2653429:::smithi18136759-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.615178871000001, 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010045017999999999, 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 63 2024-09-17T16:02:03.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883403+0000", 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:03.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883406+0000", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2320000000000001e-06 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883414+0000", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9899999999999997e-06 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883419+0000", 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5199999999999997e-06 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888936+0000", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055166759999999999 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889040+0000", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010429699999999999 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891231+0000", 2024-09-17T16:02:03.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021904110000000002 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891322+0000", 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.893447+0000", 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021253399999999999 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=18, reqid=client.4560.0:65, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.607905845000001, 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080117299999999999, 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890676+0000", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3499999999999996e-07 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890683+0000", 2024-09-17T16:02:03.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892226+0000", 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015435869999999999 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892230+0000", 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9360000000000001e-06 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892272+0000", 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1822e-05 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892278+0000", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898638+0000", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063600380000000001 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898687+0000", 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8477999999999999e-05 2024-09-17T16:02:03.566 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=4, last_complete=35'2, committed=1, applied=1))", 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.606366272999999, 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013072064, 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.567 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892217+0000", 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.063e-06 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905154+0000", 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012937357999999999 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905163+0000", 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.636e-06 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905248+0000", 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5352000000000005e-05 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905287+0000", 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8356999999999999e-05 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.571 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=133, reqid=client.4560.0:80, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.900793+0000", 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.597788117, 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061781839999999998, 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900793+0000", 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900793+0000", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900793+0000", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900804+0000", 2024-09-17T16:02:03.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0784999999999999e-05 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900890+0000", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.564e-05 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900896+0000", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1140000000000002e-06 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901752+0000", 2024-09-17T16:02:03.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00085601700000000004 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901762+0000", 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6979999999999993e-06 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.906921+0000", 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051595230000000001 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.906971+0000", 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9842000000000002e-05 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 47/32 ECSubWrite(tid=57, reqid=client.4560.0:282, at_version=47'18, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.564506253000001, 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069921009999999997, 2024-09-17T16:02:03.576 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934076+0000", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934104+0000", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8330000000000002e-05 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934155+0000", 2024-09-17T16:02:03.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1490999999999999e-05 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934165+0000", 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0121e-05 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934248+0000", 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083999999999995e-05 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934258+0000", 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0808999999999999e-05 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941026+0000", 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006767421 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941067+0000", 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0978e-05 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 47/32 ECSubWrite(tid=58, reqid=client.4560.0:279, at_version=47'22, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.934989+0000", 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.563592342, 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061212860000000001, 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934989+0000", 2024-09-17T16:02:03.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934989+0000", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934990+0000", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0139999999999999e-06 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.935046+0000", 2024-09-17T16:02:03.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6730000000000001e-05 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.935089+0000", 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3090000000000002e-05 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.935098+0000", 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0589999999999992e-06 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.935165+0000", 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6248999999999995e-05 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.935175+0000", 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0251999999999999e-05 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941085+0000", 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059102850000000004 2024-09-17T16:02:03.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941110+0000", 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4607e-05 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 47/32 ECSubWrite(tid=59, reqid=client.4560.0:280, at_version=47'23, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.562020327999999, 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074766839999999999, 2024-09-17T16:02:03.585 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936562+0000", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.64e-07 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936585+0000", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3082e-05 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936627+0000", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2141999999999997e-05 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936636+0000", 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1509999999999993e-06 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937201+0000", 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00056518400000000004 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937210+0000", 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1050000000000001e-06 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944001+0000", 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067911480000000003 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944037+0000", 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5908000000000002e-05 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.589 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 47/32 ECSubWrite(tid=60, reqid=client.4560.0:283, at_version=47'19, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.937872+0000", 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.560709070000001, 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061917200000000004, 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937872+0000", 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937872+0000", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937873+0000", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2699999999999998e-07 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937893+0000", 2024-09-17T16:02:03.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0537e-05 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937944+0000", 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1174000000000002e-05 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937953+0000", 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6680000000000007e-06 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938014+0000", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1056000000000006e-05 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938024+0000", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7569999999999999e-06 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944039+0000", 2024-09-17T16:02:03.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060150170000000001 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944064+0000", 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4683999999999999e-05 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:03.594 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.14833473, 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007077552, 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:03.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350272+0000", 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5619e-05 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350409+0000", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013671899999999999 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350418+0000", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350893+0000", 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047499899999999998 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350904+0000", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0977e-05 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357278+0000", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063733080000000003 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357324+0000", 2024-09-17T16:02:03.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6180999999999997e-05 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358797+0000", 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.139783671, 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0062492650000000004, 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358797+0000", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358797+0000", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5499999999999996e-07 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358815+0000", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6815000000000001e-05 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358867+0000", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1814000000000002e-05 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358877+0000", 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0212e-05 2024-09-17T16:02:03.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358968+0000", 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1298000000000006e-05 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358980+0000", 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1892000000000001e-05 2024-09-17T16:02:03.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365006+0000", 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00602616 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365046+0000", 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0119e-05 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 55/32 ECSubWrite(tid=97, reqid=client.4560.0:353, at_version=55'45, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.123691679, 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006702225, 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375005+0000", 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011503800000000001 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375080+0000", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5148999999999995e-05 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375088+0000", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6929999999999996e-06 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375199+0000", 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011076100000000001 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375211+0000", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1885000000000001e-05 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381558+0000", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063466750000000004 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381591+0000", 2024-09-17T16:02:03.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 64/53 ECSubWriteReply(tid=28, last_complete=64'12, committed=1, applied=1))", 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.137624191, 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79147448499999995, 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.75e-07 2024-09-17T16:02:03.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360961+0000", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9550000000000002e-06 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152290+0000", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79132856699999998 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152299+0000", 2024-09-17T16:02:03.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152377+0000", 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8317000000000002e-05 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152431+0000", 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3773000000000002e-05 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.611 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.612 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.612 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.612 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:03.614 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:03.611+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:03.714 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:03.712+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:03.721 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.721 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:03.721 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:03.721 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:03.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 3.10s0 2024-09-17T16:02:03.815 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:03.813+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:03.836 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.836 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:03.836 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:03.836 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:03.836 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:03.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:03.916 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:03.915+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:24 3.ds0 3:be8168ea:::smithi18136759-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.093969372, 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.04871255, 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.954 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 24 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6700000000000003e-07 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856438+0000", 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0089999999999999e-06 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856446+0000", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1240000000000005e-06 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856452+0000", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8490000000000002e-06 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858431+0000", 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019785380000000002 2024-09-17T16:02:03.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858505+0000", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4269000000000003e-05 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876493+0000", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017988067999999999 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876576+0000", 2024-09-17T16:02:03.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2981999999999999e-05 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905147+0000", 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028571144 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:28 3.ds0 3:b6a72c22:::smithi18136759-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.093942552000001, 2024-09-17T16:02:03.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043697429000000003, 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 28 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856462+0000", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856586+0000", 2024-09-17T16:02:03.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012445300000000001 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5877000000000001e-05 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856609+0000", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5869999999999998e-06 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858561+0000", 2024-09-17T16:02:03.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019522739999999999 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858680+0000", 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118678 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859415+0000", 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073567700000000001 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864061+0000", 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046461560000000002 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900068+0000", 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036006330000000003 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900159+0000", 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0970999999999993e-05 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:29 3.ds0 3:b6a72c22:::smithi18136759-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.093783929000001, 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043754779000000001, 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.965 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 29 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856621+0000", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2099999999999996e-07 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856624+0000", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7000000000000002e-06 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856631+0000", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856638+0000", 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859904+0000", 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032664679999999998 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859980+0000", 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5711999999999996e-05 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860364+0000", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038396600000000002 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864272+0000", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003907888 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900273+0000", 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036001247 2024-09-17T16:02:03.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900375+0000", 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101535 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:30 3.ds0 3:b6a72c22:::smithi18136759-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.093758102999999, 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.048780709999999998, 2024-09-17T16:02:03.970 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 30 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4499999999999997e-07 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856657+0000", 2024-09-17T16:02:03.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856664+0000", 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.148e-06 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856673+0000", 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.098e-06 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860632+0000", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039593440000000001 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860703+0000", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0197000000000006e-05 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900473+0000", 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039770158 2024-09-17T16:02:03.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900637+0000", 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000164356 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905426+0000", 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047893579999999996 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.975 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:40 3.ds0 3:b22f7774:::smithi18136759-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.08875475, 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039805350000000003, 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 40 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861651+0000", 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.525e-06 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861832+0000", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018106599999999999 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861846+0000", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4497e-05 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861854+0000", 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0390000000000005e-06 2024-09-17T16:02:03.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.791e-05 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862054+0000", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014139799999999999 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862812+0000", 2024-09-17T16:02:03.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000758028 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867004+0000", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004192594 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901412+0000", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034408162999999999 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901454+0000", 2024-09-17T16:02:03.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2129999999999998e-05 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:41 3.ds0 3:b22f7774:::smithi18136759-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.088528444000001, 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039711297, 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.981 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 41 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861876+0000", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4e-07 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861880+0000", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861887+0000", 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0890000000000002e-06 2024-09-17T16:02:03.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861893+0000", 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8649999999999998e-06 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863320+0000", 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001427212 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863412+0000", 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1540999999999997e-05 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863845+0000", 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043304900000000002 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867006+0000", 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031615369999999999 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901546+0000", 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034539746000000003 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901587+0000", 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0802e-05 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:42 3.ds0 3:b22f7774:::smithi18136759-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.088495949999999, 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043739746000000003, 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 42 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861909+0000", 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9400000000000005e-07 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3019999999999999e-06 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861919+0000", 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861925+0000", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8529999999999997e-06 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864137+0000", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022118649999999999 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864205+0000", 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8396e-05 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901627+0000", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037421487000000003 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901674+0000", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7577000000000002e-05 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905648+0000", 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039733169999999996 2024-09-17T16:02:03.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:52 3.ds0 3:bf204631:::smithi18136759-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.079942260999999, 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031550519999999999, 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 52 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870463+0000", 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.375e-06 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870715+0000", 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025214399999999999 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:03.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870733+0000", 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7407000000000001e-05 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870749+0000", 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6682000000000001e-05 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870815+0000", 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5939999999999995e-05 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870975+0000", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015933899999999999 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.871691+0000", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071632 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.874021+0000", 2024-09-17T16:02:03.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023304419999999998 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901948+0000", 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027927041999999999 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902012+0000", 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3829000000000002e-05 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.997 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:53 3.ds0 3:bf204631:::smithi18136759-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870774+0000", 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.079629800999999, 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031465389000000003, 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:03.998 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 53 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870774+0000", 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870774+0000", 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:03.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870775+0000", 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5199999999999996e-07 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870780+0000", 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0409999999999997e-06 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870787+0000", 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6599999999999995e-06 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870793+0000", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6180000000000003e-06 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.872307+0000", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001514325 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.872391+0000", 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3808999999999997e-05 2024-09-17T16:02:04.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.872781+0000", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00039004699999999997 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.874024+0000", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012423079999999999 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902151+0000", 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028127421 2024-09-17T16:02:04.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902239+0000", 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8508e-05 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:54 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.079601142000001, 2024-09-17T16:02:04.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035090831000000003, 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 54 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9800000000000004e-07 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870806+0000", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3549999999999999e-06 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870813+0000", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1539999999999996e-06 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870819+0000", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0700000000000003e-06 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873071+0000", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022519200000000001 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873154+0000", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3746999999999996e-05 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902680+0000", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029525513 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902767+0000", 2024-09-17T16:02:04.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6837999999999998e-05 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905894+0000", 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031267360000000002 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=2, last_complete=35'3, committed=1, applied=1))", 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875920+0000", 2024-09-17T16:02:04.008 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.074483613999998, 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024249401, 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875920+0000", 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875920+0000", 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875921+0000", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2300000000000001e-07 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875924+0000", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0230000000000001e-06 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876138+0000", 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000214305 2024-09-17T16:02:04.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876143+0000", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.741e-06 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.899982+0000", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023839477000000001 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900170+0000", 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018733199999999999 2024-09-17T16:02:04.011 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875926+0000", 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.074477734999999, 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024475957999999999, 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875926+0000", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875926+0000", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875926+0000", 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.72e-07 2024-09-17T16:02:04.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875928+0000", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876154+0000", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000226336 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876158+0000", 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1110000000000001e-06 2024-09-17T16:02:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900200+0000", 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024041665 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900402+0000", 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000202201 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 35/32 ECSubWriteReply(tid=4, last_complete=35'5, committed=1, applied=1))", 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.876835+0000", 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.073568649999999, 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024220789, 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876835+0000", 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876835+0000", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876836+0000", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2900000000000003e-07 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876841+0000", 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8330000000000001e-06 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876876+0000", 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5448999999999999e-05 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876883+0000", 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7780000000000002e-06 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901036+0000", 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.024153355000000001 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901056+0000", 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9644999999999999e-05 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:70 3.ds0 3:b27fc74f:::smithi18136759-24:head [append 0~393216,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.885204+0000", 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.065200111999999, 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027370377000000001, 2024-09-17T16:02:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 70 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885204+0000", 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885204+0000", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885205+0000", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5910000000000001e-06 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885539+0000", 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033399699999999999 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885555+0000", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5960999999999998e-05 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885574+0000", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9011000000000001e-05 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905940+0000", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020365477999999999 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.906093+0000", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015314899999999999 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.907023+0000", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00093029799999999998 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.911540+0000", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045168420000000001 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912496+0000", 2024-09-17T16:02:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095607899999999998 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912574+0000", 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7971000000000003e-05 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:71 3.ds0 3:b27fc74f:::smithi18136759-24:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.885596+0000", 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.064808371000002, 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027182386999999999, 2024-09-17T16:02:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 71 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885596+0000", 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885596+0000", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885596+0000", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885602+0000", 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4399999999999996e-06 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885612+0000", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0189e-05 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885618+0000", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7299999999999999e-06 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.907512+0000", 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.021893780000000002 2024-09-17T16:02:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.907600+0000", 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7627999999999995e-05 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.908031+0000", 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043151799999999998 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.911541+0000", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035098809999999999 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912711+0000", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011700759999999999 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912778+0000", 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6538000000000006e-05 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:72 3.ds0 3:b27fc74f:::smithi18136759-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.885639+0000", 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 30.064764527000001, 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028004828999999998, 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 72 2024-09-17T16:02:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885639+0000", 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885639+0000", 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885640+0000", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4600000000000005e-07 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885643+0000", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0280000000000001e-06 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885653+0000", 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0172e-05 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.885661+0000", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0260000000000007e-06 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.908305+0000", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022643735000000002 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.908390+0000", 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4758000000000004e-05 2024-09-17T16:02:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912817+0000", 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044273480000000002 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.912904+0000", 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7305000000000005e-05 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.913644+0000", 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000739911 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 46/32 ECSubWriteReply(tid=54, last_complete=46'16, committed=1, applied=1))", 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.011554926999999, 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62914934300000003, 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938850+0000", 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.06e-07 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938855+0000", 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0010000000000001e-06 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567797+0000", 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62894212500000002 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567814+0000", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6847e-05 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567931+0000", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011766500000000001 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567998+0000", 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6999e-05 2024-09-17T16:02:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:284 3.as0 3:5583c93f:::smithi18136759-40:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.016885948999999, 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.386964417, 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 284 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933519+0000", 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933522+0000", 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4010000000000001e-06 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933536+0000", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4269e-05 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933544+0000", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.576e-06 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938388+0000", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048441300000000003 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938478+0000", 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9975000000000005e-05 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939194+0000", 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071552499999999995 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944047+0000", 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048533990000000004 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320427+0000", 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37638003799999997 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320482+0000", 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5390000000000003e-05 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:358 3.ds0 3:b27fc74f:::smithi18136759-24:head [copy-from ver 28] snapc 11=[11,10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.574287255, 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.040742296999999997, 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 358 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376121+0000", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1679999999999997e-06 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376135+0000", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3054000000000001e-05 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376141+0000", 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.443e-06 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381260+0000", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005118671 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381327+0000", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7288999999999999e-05 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.392972+0000", 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011645071999999999 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.396350+0000", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033780709999999999 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416805+0000", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020454785999999999 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416859+0000", 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4045999999999997e-05 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 63/32 ECSubWriteReply(tid=124, last_complete=63'41, committed=1, applied=1))", 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 2.523075167, 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45322852000000002, 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427336+0000", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4069999999999998e-06 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880036+0000", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45270011199999999 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880042+0000", 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7950000000000004e-06 2024-09-17T16:02:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880109+0000", 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7659999999999999e-05 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880557+0000", 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044786499999999998 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:04.053 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:04.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:04.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:04.056 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:04.053+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:04.156 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:04.154+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:04.176 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:04.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:04.228 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:04.228 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:04.249 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:04.250 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:04.257 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:04.255+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:04.271 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:04.271 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:04.293 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:04.293 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:04.293 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.294 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:04.313 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:04.313 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:04.334 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:04.335 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:04.338 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:181: done (1 left) 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:182: rmattr oid 1 current snap is 22 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:182: done (1 left) 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:183: read oid 7 snap 14 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:183: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-17T16:02:04.342 INFO:tasks.rados.rados.0.smithi181.stdout:184: snap_remove snap 8 2024-09-17T16:02:04.359 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:04.356+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:04.363 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:04.364 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:04.384 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:04.384 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:04.407 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:04.407 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:04.419 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.419 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:04.428 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:04.428 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:04.450 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:04.450 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:04.459 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:04.457+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:04.471 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:04.471 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:04.491 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:04.492 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:04.513 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:04.513 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:04.534 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:04.532+0000 7fa139b3b640 0 -- 172.21.15.6:0/3803658088 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fa134221d00 msgr2=0x7fa1342240f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:04.535 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.535 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.540 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:04.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:04.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:04.587 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:04.585+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:04.652 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.652 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:04.661 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:04.658+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:04.761 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:04.759+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.767 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:04.862 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:04.861+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:04.923 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:04.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:04.963 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:04.962+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.039 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:05.064 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:05.063+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:05.155 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.155 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:05.166 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:05.164+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:05.226 INFO:teuthology.orchestra.run.smithi181.stdout: marking collection for removal 2024-09-17T16:02:05.226 INFO:teuthology.orchestra.run.smithi181.stdout:setting '_remove' omap key 2024-09-17T16:02:05.226 INFO:teuthology.orchestra.run.smithi181.stdout:finish_remove_pgs 3.10s0_head removing 3.10s0 2024-09-17T16:02:05.232 INFO:teuthology.orchestra.run.smithi181.stdout:Remove successful 2024-09-17T16:02:05.266 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:05.264+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:05.270 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.270 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:05.343 INFO:tasks.rados.rados.0.smithi181.stdout:184: done (2 left) 2024-09-17T16:02:05.343 INFO:tasks.rados.rados.0.smithi181.stdout:185: read oid 41 snap -1 2024-09-17T16:02:05.343 INFO:tasks.rados.rados.0.smithi181.stdout:185: expect deleted 2024-09-17T16:02:05.343 INFO:tasks.rados.rados.0.smithi181.stdout:186: snap_remove snap 0 2024-09-17T16:02:05.367 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:05.365+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:05.393 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.393 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.401 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:05.467 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:05.465+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:05.549 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.549 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.558 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:05.568 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:05.567+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:05.669 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:05.668+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:05.672 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.672 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:05.742 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.10s0.7 2024-09-17T16:02:05.770 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:05.768+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:05.789 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.789 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.794 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:05.871 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:05.869+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:05.897 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:05.906 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:05.972 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:05.970+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:06.010 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.010 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:06.073 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:06.071+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:06.162 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.162 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.170 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:06.173 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:06.171+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:06.274 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.274 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.274 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:06.272+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:06.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:06.346 INFO:tasks.rados.rados.0.smithi181.stdout:186: done (3 left) 2024-09-17T16:02:06.346 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 30 current snap is 22 2024-09-17T16:02:06.346 INFO:tasks.rados.rados.0.smithi181.stdout:187: seq_num 85 ranges {3047424=671744,3719168=770048} 2024-09-17T16:02:06.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:02:06.367 INFO:tasks.rados.rados.0.smithi181.stdout:187: writing smithi18136759-30 from 3047424 to 3719168 tid 1 2024-09-17T16:02:06.372 INFO:tasks.rados.rados.0.smithi181.stdout:187: writing smithi18136759-30 from 3719168 to 4489216 tid 2 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:188: read oid 29 snap 10 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:188: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:189: rollback oid 17 current snap is 22 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 17 to 19 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:183: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 97754 2024-09-17T16:02:06.373 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:06.374 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 41111 2024-09-17T16:02:06.375 INFO:tasks.rados.rados.0.smithi181.stdout:179: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-17T16:02:06.376 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 227309 2024-09-17T16:02:06.377 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 79995 2024-09-17T16:02:06.379 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 74313 2024-09-17T16:02:06.381 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing rollback tid 0 to smithi18136759-17 2024-09-17T16:02:06.382 INFO:tasks.rados.rados.0.smithi181.stdout:179: done (5 left) 2024-09-17T16:02:06.382 INFO:tasks.rados.rados.0.smithi181.stdout:183: done (4 left) 2024-09-17T16:02:06.382 INFO:tasks.rados.rados.0.smithi181.stdout:185: done (3 left) 2024-09-17T16:02:06.382 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 10 current snap is 22 2024-09-17T16:02:06.382 INFO:tasks.rados.rados.0.smithi181.stdout:190: seq_num 86 ranges {442368=688128,1130496=540672,1671168=229376} 2024-09-17T16:02:06.388 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18136759-10 from 442368 to 1130496 tid 1 2024-09-17T16:02:06.391 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18136759-10 from 1130496 to 1671168 tid 2 2024-09-17T16:02:06.393 INFO:tasks.rados.rados.0.smithi181.stdout:190: writing smithi18136759-10 from 1671168 to 1900544 tid 3 2024-09-17T16:02:06.393 INFO:tasks.rados.rados.0.smithi181.stdout:191: delete oid 36 current snap is 22 2024-09-17T16:02:06.393 INFO:tasks.rados.rados.0.smithi181.stdout:187: finishing write tid 1 to smithi18136759-30 2024-09-17T16:02:06.393 INFO:tasks.rados.rados.0.smithi181.stdout:187: finishing write tid 2 to smithi18136759-30 2024-09-17T16:02:06.393 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing rollback tid 1 to smithi18136759-17 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: finishing write tid 3 to smithi18136759-30 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:189: finishing rollback tid 2 to smithi18136759-17 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 33 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: finishing write tid 4 to smithi18136759-30 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: oid 30 updating version 0 to 22 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: oid 30 updating version 22 to 24 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: oid 30 version 24 is already newer than 23 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 30 v 24 (ObjNum 85 snap 22 seq_num 85) dirty exists 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:187: left oid 30 (ObjNum 85 snap 22 seq_num 85) 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:188: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-17T16:02:06.394 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:06.399 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 412659 2024-09-17T16:02:06.400 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:06.403 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:06.401+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 1 to smithi18136759-10 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 2 to smithi18136759-10 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 3 to smithi18136759-10 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 4 to smithi18136759-10 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: finishing write tid 5 to smithi18136759-10 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 10 updating version 0 to 98 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 10 updating version 98 to 99 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 10 version 99 is already newer than 97 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: oid 10 version 99 is already newer than 96 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 99 (ObjNum 86 snap 22 seq_num 86) dirty exists 2024-09-17T16:02:06.405 INFO:tasks.rados.rados.0.smithi181.stdout:190: left oid 10 (ObjNum 86 snap 22 seq_num 86) 2024-09-17T16:02:06.436 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.436 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:06.489 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:06.487+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:06.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:06.549 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.549 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.557 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:06.591 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:06.589+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:06.677 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:06.675+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:06.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:06.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:06.702 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.702 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.703 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:06.703 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:06.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:06.725 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:06.725 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:06.747 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:06.747 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:06.770 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:06.770 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:06.778 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:06.776+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:06.791 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:06.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:06.814 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:06.814 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:06.814 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:06.834 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:06.835 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:187: done (4 left) 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:188: done (3 left) 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:189: done (2 left) 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:190: done (1 left) 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:191: done (0 left) 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 44 current snap is 22 2024-09-17T16:02:06.839 INFO:tasks.rados.rados.0.smithi181.stdout:192: seq_num 88 ranges {1736704=720896,2457600=442368,2899968=327680} 2024-09-17T16:02:06.852 INFO:tasks.rados.rados.0.smithi181.stdout:192: writing smithi18136759-44 from 1736704 to 2457600 tid 1 2024-09-17T16:02:06.854 INFO:tasks.rados.rados.0.smithi181.stdout:192: writing smithi18136759-44 from 2457600 to 2899968 tid 2 2024-09-17T16:02:06.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:06.856 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:06.857 INFO:tasks.rados.rados.0.smithi181.stdout:192: writing smithi18136759-44 from 2899968 to 3227648 tid 3 2024-09-17T16:02:06.857 INFO:tasks.rados.rados.0.smithi181.stdout:193: read oid 3 snap -1 2024-09-17T16:02:06.857 INFO:tasks.rados.rados.0.smithi181.stdout:193: expect (ObjNum 62 snap 8 seq_num 62) 2024-09-17T16:02:06.857 INFO:tasks.rados.rados.0.smithi181.stdout:194: snap_remove snap 19 2024-09-17T16:02:06.858 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:06.858 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:06.869 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:06.879 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:06.880 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:06.877+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:06.901 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:06.901 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:06.921 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:06.921 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:06.943 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:06.943 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:06.964 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:06.965 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:06.970 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:06.968+0000 7fa8037fe640 0 -- 172.21.15.6:0/2885673542 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fa80421f910 msgr2=0x7fa804221d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:06.971 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:06.972 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:06.972 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:06.972 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:06.972 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:06.980 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:06.981 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:06.979+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:06.991 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:06.991 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:06.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:07.080 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:07.079+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.229450579000002, 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:07.085 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:07.086 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.087 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.088 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.089 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.229422065999998, 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.090 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.091 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:02:07.092 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:02:07.093 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:02:07.094 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.095 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.229283021000001, 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:02:07.096 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.097 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.098 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:02:07.099 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:07.100 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:07.101 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.229255346999999, 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.102 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.103 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:02:07.104 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:02:07.105 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.106 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=2, last_complete=34'3, committed=1, applied=1))", 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.218679102999999, 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014226245, 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.107 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863555+0000", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7500000000000001e-07 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863557+0000", 2024-09-17T16:02:07.108 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1169999999999998e-06 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863638+0000", 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.03e-05 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863644+0000", 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4150000000000004e-06 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.109 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877622+0000", 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013978305 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877781+0000", 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015883299999999999 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.110 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 34/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.218675433000001, 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014337127, 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.111 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863558+0000", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863559+0000", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5800000000000001e-07 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863560+0000", 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7600000000000001e-06 2024-09-17T16:02:07.112 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863674+0000", 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011379000000000001 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863682+0000", 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4139999999999997e-06 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.113 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877875+0000", 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014193012 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877896+0000", 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0993e-05 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s4 35/32 ECSubWrite(tid=119, reqid=client.4560.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:02:07.114 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.217536612000004, 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015598302, 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864697+0000", 2024-09-17T16:02:07.115 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864698+0000", 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.44e-07 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864708+0000", 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.73e-06 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.116 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864735+0000", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7804e-05 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864741+0000", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7969999999999997e-06 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877824+0000", 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01308231 2024-09-17T16:02:07.117 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877833+0000", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4809999999999995e-06 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880272+0000", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024384799999999998 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880296+0000", 2024-09-17T16:02:07.118 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4056000000000001e-05 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.164818769, 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.119 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:07.120 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:02:07.121 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.122 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:07.123 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:07.124 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.164793564999997, 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.125 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.126 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:02:07.127 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:02:07.128 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.129 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.161427570999997, 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.130 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:02:07.131 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:02:07.132 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.134452922000001, 2024-09-17T16:02:07.133 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.134 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:02:07.135 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.136 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.137 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.138 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.134435050999997, 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.139 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.140 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.141 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:02:07.142 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.143 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.131141284000002, 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.144 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:02:07.147 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:185 3.9s0 3:99c5a4ac:::smithi18136759-42:head [copy-from ver 7] snapc 4=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 29.209303076000001, 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.019207894999999999, 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.148 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 185 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872931+0000", 2024-09-17T16:02:07.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872932+0000", 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.635e-06 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872938+0000", 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1680000000000003e-06 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872948+0000", 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0477e-05 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872955+0000", 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7750000000000004e-06 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873015+0000", 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9954000000000002e-05 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873024+0000", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9439999999999994e-06 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874582+0000", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00155798 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874682+0000", 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000100215 2024-09-17T16:02:07.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.887716+0000", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013033741999999999 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889595+0000", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0018794879999999999 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892089+0000", 2024-09-17T16:02:07.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0024933450000000001 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.892139+0000", 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0172e-05 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:186 3.9s0 3:99c5a4ac:::smithi18136759-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e39)", 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 29.209266441, 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016243253999999999, 2024-09-17T16:02:07.154 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 186 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872967+0000", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872968+0000", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2500000000000001e-07 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872969+0000", 2024-09-17T16:02:07.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.525e-06 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872975+0000", 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.186e-06 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.872978+0000", 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.202e-06 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873075+0000", 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7590999999999994e-05 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.873079+0000", 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.54e-06 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874867+0000", 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0017885100000000001 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.158 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.874930+0000", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2175999999999996e-05 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889013+0000", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014083304 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889112+0000", 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8972e-05 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:37.889211+0000", 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8923000000000001e-05 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.131025528999999, 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.160 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:02:07.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.151172149000001, 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.164 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:07.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 55/53 ECSubWriteReply(tid=63, last_complete=55'29, committed=1, applied=1))", 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.705256909999999, 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.850069713, 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376978+0000", 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376981+0000", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226831+0000", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84985070900000004 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226846+0000", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4192999999999999e-05 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226976+0000", 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013074699999999999 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.227047+0000", 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0347999999999998e-05 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 55/32 ECSubWrite(tid=100, reqid=client.4560.0:356, at_version=55'48, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.380903+0000", 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.701330706, 2024-09-17T16:02:07.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013579861, 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.380903+0000", 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.380903+0000", 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.380904+0000", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2499999999999995e-07 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.380917+0000", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2802e-05 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393985+0000", 2024-09-17T16:02:07.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013068079 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393990+0000", 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0880000000000002e-06 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.394026+0000", 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6477000000000003e-05 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.394032+0000", 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2910000000000004e-06 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.394449+0000", 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00041635199999999999 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.394483+0000", 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4246999999999999e-05 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.175 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.689057414000001, 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023393622999999999, 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:07.176 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.82e-07 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393970+0000", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079326400000000001 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414706+0000", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020736065000000001 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414712+0000", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:07.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414788+0000", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6206999999999997e-05 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414798+0000", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416538+0000", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00174007 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416570+0000", 2024-09-17T16:02:07.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2415000000000001e-05 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:07.180 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:07.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:07.182 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:07.179+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:07.193 INFO:teuthology.orchestra.run.smithi181.stdout:get_pg_num_history pg_num_history pg_num_history(e64 pg_nums {1={23=1},2={25=8},3={32=16,53=26}} deleted_pools ) 2024-09-17T16:02:07.193 INFO:teuthology.orchestra.run.smithi181.stdout:Importing pgid 3.10s0 2024-09-17T16:02:07.193 INFO:teuthology.orchestra.run.smithi181.stdout:Write 0#3:0a7b2816:::smithi18136759-31:9# 2024-09-17T16:02:07.194 INFO:teuthology.orchestra.run.smithi181.stdout:Write 0#3:0a7b2816:::smithi18136759-31:11# 2024-09-17T16:02:07.196 INFO:teuthology.orchestra.run.smithi181.stdout:Write 0#3:0a7b2816:::smithi18136759-31:head# 2024-09-17T16:02:07.196 INFO:teuthology.orchestra.run.smithi181.stdout:snapset 11={9=[9,8,6,2],11=[10,e,d,c,a]} 2024-09-17T16:02:07.197 INFO:teuthology.orchestra.run.smithi181.stdout:write_pg epoch 64 info 3.10s0( v 64'35 (0'0,64'35] local-lis/les=57/58 n=3 ec=53/32 lis/c=57/57 les/c/f=58/58/0 sis=57) 2024-09-17T16:02:07.197 INFO:teuthology.orchestra.run.smithi181.stdout:Import successful 2024-09-17T16:02:07.282 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:07.280+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:07.285 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:07.285 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:07.285 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:07.285 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:07.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:07.335 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:02:07.341 INFO:tasks.rados.rados.0.smithi181.stdout:194: done (2 left) 2024-09-17T16:02:07.341 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 1 current snap is 22 2024-09-17T16:02:07.341 INFO:tasks.rados.rados.0.smithi181.stdout:195: seq_num 89 ranges {0=737280,737280=573440,1310720=196608} 2024-09-17T16:02:07.345 INFO:tasks.rados.rados.0.smithi181.stdout:195: writing smithi18136759-1 from 0 to 737280 tid 1 2024-09-17T16:02:07.349 INFO:tasks.rados.rados.0.smithi181.stdout:195: writing smithi18136759-1 from 737280 to 1310720 tid 2 2024-09-17T16:02:07.351 INFO:tasks.rados.rados.0.smithi181.stdout:195: writing smithi18136759-1 from 1310720 to 1507328 tid 3 2024-09-17T16:02:07.351 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 45 current snap is 22 2024-09-17T16:02:07.351 INFO:tasks.rados.rados.0.smithi181.stdout:196: seq_num 90 ranges {753664=753664,1507328=720896,2228224=49152} 2024-09-17T16:02:07.360 INFO:tasks.rados.rados.0.smithi181.stdout:196: writing smithi18136759-45 from 753664 to 1507328 tid 1 2024-09-17T16:02:07.364 INFO:tasks.rados.rados.0.smithi181.stdout:196: writing smithi18136759-45 from 1507328 to 2228224 tid 2 2024-09-17T16:02:07.365 INFO:tasks.rados.rados.0.smithi181.stdout:196: writing smithi18136759-45 from 2228224 to 2277376 tid 3 2024-09-17T16:02:07.365 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 28 current snap is 22 2024-09-17T16:02:07.365 INFO:tasks.rados.rados.0.smithi181.stdout:192: finishing write tid 1 to smithi18136759-44 2024-09-17T16:02:07.366 INFO:tasks.rados.rados.0.smithi181.stdout:192: finishing write tid 2 to smithi18136759-44 2024-09-17T16:02:07.366 INFO:tasks.rados.rados.0.smithi181.stdout:197: seq_num 91 ranges {0=770048,770048=475136,1245184=294912} 2024-09-17T16:02:07.370 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18136759-28 from 0 to 770048 tid 1 2024-09-17T16:02:07.372 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18136759-28 from 770048 to 1245184 tid 2 2024-09-17T16:02:07.374 INFO:tasks.rados.rados.0.smithi181.stdout:197: writing smithi18136759-28 from 1245184 to 1540096 tid 3 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:198: rollback oid 13 current snap is 22 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 13 to 4 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: finishing write tid 3 to smithi18136759-44 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: finishing write tid 4 to smithi18136759-44 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: finishing write tid 5 to smithi18136759-44 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: oid 44 updating version 0 to 48 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: oid 44 version 48 is already newer than 47 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: oid 44 version 48 is already newer than 46 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: oid 44 version 48 is already newer than 45 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 48 (ObjNum 88 snap 22 seq_num 88) dirty exists 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:192: left oid 44 (ObjNum 88 snap 22 seq_num 88) 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:195: finishing write tid 1 to smithi18136759-1 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:193: expect (ObjNum 62 snap 8 seq_num 62) 2024-09-17T16:02:07.375 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:07.383 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:07.381+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:07.387 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 174675 2024-09-17T16:02:07.395 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:195: finishing write tid 2 to smithi18136759-1 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:195: finishing write tid 3 to smithi18136759-1 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:192: done (5 left) 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:193: done (4 left) 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:199: rmattr oid 35 current snap is 22 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:195: finishing write tid 4 to smithi18136759-1 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:196: finishing write tid 1 to smithi18136759-45 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:196: finishing write tid 2 to smithi18136759-45 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:196: finishing write tid 3 to smithi18136759-45 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:195: finishing write tid 5 to smithi18136759-1 2024-09-17T16:02:07.407 INFO:tasks.rados.rados.0.smithi181.stdout:195: oid 1 updating version 0 to 37 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:195: oid 1 version 37 is already newer than 35 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:195: oid 1 version 37 is already newer than 34 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:195: oid 1 version 37 is already newer than 36 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 37 (ObjNum 89 snap 22 seq_num 89) dirty exists 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:195: left oid 1 (ObjNum 89 snap 22 seq_num 89) 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 1 to smithi18136759-28 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 2 to smithi18136759-28 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:198: finishing rollback tid 0 to smithi18136759-13 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:198: finishing rollback tid 1 to smithi18136759-13 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 3 to smithi18136759-28 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 4 to smithi18136759-28 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: finishing write tid 5 to smithi18136759-28 2024-09-17T16:02:07.408 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 28 updating version 0 to 34 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 28 version 34 is already newer than 33 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 28 updating version 34 to 36 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:197: oid 28 version 36 is already newer than 35 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 36 (ObjNum 91 snap 22 seq_num 91) dirty exists 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:197: left oid 28 (ObjNum 91 snap 22 seq_num 91) 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:195: done (4 left) 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:197: done (3 left) 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:200: rmattr oid 16 current snap is 22 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:200: done (3 left) 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:201: delete oid 41 current snap is 22 2024-09-17T16:02:07.409 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 0 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:199: done (3 left) 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:201: done (2 left) 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:202: rollback oid 3 current snap is 22 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 3 to 11 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:203: delete oid 32 current snap is 22 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:198: finishing rollback tid 2 to smithi18136759-13 2024-09-17T16:02:07.410 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 23 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-17T16:02:07.413 INFO:tasks.rados.rados.0.smithi181.stdout:198: done (3 left) 2024-09-17T16:02:07.413 INFO:tasks.rados.rados.0.smithi181.stdout:203: done (2 left) 2024-09-17T16:02:07.413 INFO:tasks.rados.rados.0.smithi181.stdout:204: snap_create 2024-09-17T16:02:07.415 INFO:tasks.rados.rados.0.smithi181.stdout:202: finishing rollback tid 0 to smithi18136759-3 2024-09-17T16:02:07.415 INFO:tasks.rados.rados.0.smithi181.stdout:202: finishing rollback tid 1 to smithi18136759-3 2024-09-17T16:02:07.418 INFO:tasks.rados.rados.0.smithi181.stdout:202: finishing rollback tid 2 to smithi18136759-3 2024-09-17T16:02:07.418 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 50 (ObjNum 62 snap 8 seq_num 62) dirty exists 2024-09-17T16:02:07.450 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:07.450 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:07.450 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:07.451 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:07.451 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:07.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:07.483 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:07.482+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:07.584 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:07.582+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:07.603 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=167, reqid=client.4560.0:228, at_version=44'14, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.692804409000001, 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045964070000000003, 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.604 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908482+0000", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908499+0000", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6748999999999998e-05 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908533+0000", 2024-09-17T16:02:07.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3831999999999997e-05 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908539+0000", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5980000000000002e-06 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908619+0000", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740999999999994e-05 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908629+0000", 2024-09-17T16:02:07.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913043+0000", 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044149380000000002 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913078+0000", 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4094999999999998e-05 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.673088235000002, 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003430879, 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928198+0000", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2200000000000002e-07 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928663+0000", 2024-09-17T16:02:07.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046509200000000001 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928719+0000", 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5677000000000002e-05 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928728+0000", 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6729999999999998e-06 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928810+0000", 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2386999999999996e-05 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928821+0000", 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0797e-05 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931371+0000", 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002550549 2024-09-17T16:02:07.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931628+0000", 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000256782 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.673541228000001, 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046997530000000001, 2024-09-17T16:02:07.613 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927762+0000", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7229000000000001e-05 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927945+0000", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018362200000000001 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928161+0000", 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021584100000000001 2024-09-17T16:02:07.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928170+0000", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6249999999999996e-06 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928242+0000", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2434999999999993e-05 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928253+0000", 2024-09-17T16:02:07.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932409+0000", 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041564169999999999 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932444+0000", 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.673336202000002, 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048629329999999998, 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927950+0000", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3399999999999999e-07 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928041+0000", 2024-09-17T16:02:07.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0953999999999997e-05 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928205+0000", 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000164537 2024-09-17T16:02:07.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928213+0000", 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.272e-06 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929483+0000", 2024-09-17T16:02:07.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001270106 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929493+0000", 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0207000000000001e-05 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932778+0000", 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032847419999999998 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932812+0000", 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4581e-05 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=12, reqid=client.4560.0:273, at_version=45'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.673204683000002, 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043524990000000001, 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.623 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928082+0000", 2024-09-17T16:02:07.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2799999999999996e-07 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928098+0000", 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6070999999999999e-05 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928229+0000", 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013130999999999999 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928234+0000", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928583+0000", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034869300000000002 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928595+0000", 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1433e-05 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932398+0000", 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038034890000000002 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932433+0000", 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5386999999999998e-05 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:07.627 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.672544296000002, 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003718285, 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928742+0000", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928761+0000", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8666000000000001e-05 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928789+0000", 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8535999999999999e-05 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928798+0000", 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4479999999999994e-06 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928947+0000", 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000149068 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928956+0000", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0799999999999995e-06 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932437+0000", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034817609999999999 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932460+0000", 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2078000000000001e-05 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.631 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.682742946000001, 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039399220000000002, 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.29e-07 2024-09-17T16:02:07.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5406999999999999e-05 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102031 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918679+0000", 2024-09-17T16:02:07.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918809+0000", 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012967600000000001 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918820+0000", 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1156e-05 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922452+0000", 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036319080000000001 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922483+0000", 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0338999999999999e-05 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:07.636 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.682316847999999, 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035116600000000002, 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918994+0000", 2024-09-17T16:02:07.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4694000000000001e-05 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919035+0000", 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1223000000000002e-05 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919044+0000", 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0599999999999997e-06 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919108+0000", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3326999999999999e-05 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919118+0000", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0037999999999999e-05 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922439+0000", 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033215340000000001 2024-09-17T16:02:07.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922480+0000", 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1010000000000002e-05 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.681687999000001, 2024-09-17T16:02:07.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043836329999999996, 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919621+0000", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2204e-05 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919652+0000", 2024-09-17T16:02:07.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1846e-05 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919661+0000", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3075000000000002e-05 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919733+0000", 2024-09-17T16:02:07.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3540000000000006e-06 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923937+0000", 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042032980000000003 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923981+0000", 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4685000000000003e-05 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.645 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=79, reqid=client.4560.0:332, at_version=52'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.243671170999999, 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051054819999999997, 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:07.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357615+0000", 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-07 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358043+0000", 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042830599999999999 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358075+0000", 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2512999999999999e-05 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358085+0000", 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0081000000000001e-05 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358170+0000", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4362000000000003e-05 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358181+0000", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1535000000000001e-05 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362680+0000", 2024-09-17T16:02:07.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989069999999999 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362720+0000", 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9641000000000002e-05 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=81, reqid=client.4560.0:334, at_version=52'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:07.650 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 17.238505331999999, 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033175610000000001, 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362781+0000", 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.71e-07 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362824+0000", 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3049000000000001e-05 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362863+0000", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9063999999999997e-05 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362873+0000", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4199999999999996e-06 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.363453+0000", 2024-09-17T16:02:07.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057987999999999996 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.363463+0000", 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0287e-05 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.366063+0000", 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026005630000000002 2024-09-17T16:02:07.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.366098+0000", 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4427000000000001e-05 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=197, reqid=client.4560.0:392, at_version=60'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:07.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.2509136630000004, 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0099856809999999997, 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2599999999999999e-07 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350537+0000", 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016478400000000001 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350604+0000", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7305999999999996e-05 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350615+0000", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1154e-05 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350701+0000", 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6136000000000003e-05 2024-09-17T16:02:07.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350721+0000", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9967e-05 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360314+0000", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0095931230000000003 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360358+0000", 2024-09-17T16:02:07.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3084999999999998e-05 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=198, reqid=client.4560.0:393, at_version=60'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.2483879469999994, 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075206600000000002, 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352899+0000", 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1230000000000001e-06 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353041+0000", 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014228299999999999 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353091+0000", 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9942999999999999e-05 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353101+0000", 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353167+0000", 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5365999999999997e-05 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353178+0000", 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1429e-05 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360364+0000", 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071857500000000003 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360418+0000", 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4317000000000002e-05 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=199, reqid=client.4560.0:394, at_version=60'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.2466993960000003, 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058628329999999996, 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354587+0000", 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4099999999999997e-07 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354652+0000", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5085000000000005e-05 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354702+0000", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9641999999999997e-05 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354711+0000", 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0869999999999996e-06 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354774+0000", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3048999999999999e-05 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354784+0000", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163000000000001e-05 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360427+0000", 2024-09-17T16:02:07.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056429729999999999 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360449+0000", 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1993000000000001e-05 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=200, reqid=client.4560.0:395, at_version=60'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:07.669 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.2452732569999991, 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989890000000001, 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:07.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356013+0000", 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356031+0000", 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7592999999999999e-05 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356068+0000", 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7345999999999998e-05 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356079+0000", 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0730999999999999e-05 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356150+0000", 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1668999999999994e-05 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356161+0000", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1027e-05 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360478+0000", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043165800000000004 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360511+0000", 2024-09-17T16:02:07.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3251999999999999e-05 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2630295440000001, 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046971620000000004, 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.674 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:07.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338276+0000", 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9932000000000002e-05 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338298+0000", 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1336e-05 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338306+0000", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6810000000000004e-06 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338467+0000", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160538 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338477+0000", 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5929999999999995e-06 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342913+0000", 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044366149999999997 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342953+0000", 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0061000000000002e-05 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=134, reqid=osd.13.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=64'15))", 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2512387839999999, 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042510439999999998, 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350056+0000", 2024-09-17T16:02:07.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350080+0000", 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4360999999999999e-05 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350089+0000", 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2609999999999993e-06 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350189+0000", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100868 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350199+0000", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1679999999999993e-06 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354267+0000", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040685629999999999 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354298+0000", 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0626000000000001e-05 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 64/53 ECSubWrite(tid=211, reqid=osd.11.0:211, at_version=64'37, trim_to=0'0, roll_forward_to=62'35))", 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 5.2422194119999999, 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048593489999999998, 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359067+0000", 2024-09-17T16:02:07.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6699999999999999e-07 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359087+0000", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0594999999999998e-05 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359129+0000", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1458999999999997e-05 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359138+0000", 2024-09-17T16:02:07.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6050000000000004e-06 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359207+0000", 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8885000000000002e-05 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359218+0000", 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0524e-05 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363901+0000", 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046832100000000002 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363925+0000", 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4703999999999999e-05 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.250260765, 2024-09-17T16:02:07.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059954810000000004, 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351050+0000", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4579e-05 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351200+0000", 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015020800000000001 2024-09-17T16:02:07.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351208+0000", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6799999999999999e-06 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352434+0000", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012252350000000001 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352445+0000", 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1440000000000001e-05 2024-09-17T16:02:07.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.356973+0000", 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045277900000000003 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.357020+0000", 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=102, reqid=client.4560.0:502, at_version=69'50, trim_to=0'0, roll_forward_to=69'46))", 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.19127977700000001, 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042988599999999998, 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.37e-07 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410021+0000", 2024-09-17T16:02:07.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4741e-05 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410042+0000", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0798e-05 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410047+0000", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5459999999999999e-06 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410084+0000", 2024-09-17T16:02:07.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6556e-05 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410089+0000", 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4600000000000002e-06 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414267+0000", 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041775720000000001 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:07.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414305+0000", 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.765e-05 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.696 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:07.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:07.698 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:07.696+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:07.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f /var/log/ceph/exp.3.10s0.7 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:488 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [setxattr _header (57) in=64b] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.43788801999999999, 2024-09-17T16:02:07.803 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.437915585, 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 488 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:07.804 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.805 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364123+0000", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9220000000000001e-06 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364131+0000", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364137+0000", 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4679999999999997e-06 2024-09-17T16:02:07.806 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366664+0000", 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025266949999999998 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366744+0000", 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0739000000000001e-05 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:07.807 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367168+0000", 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042360999999999999 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368066+0000", 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089781599999999996 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.808 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:489 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.43786180000000002, 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43793210399999999, 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "waiting for rw locks", 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 489 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:07.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364147+0000", 2024-09-17T16:02:07.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364149+0000", 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2359999999999999e-06 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364156+0000", 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4630000000000004e-06 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364163+0000", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368609+0000", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044454869999999997 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368658+0000", 2024-09-17T16:02:07.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9438999999999999e-05 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 2 2024-09-17T16:02:07.813 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:07.813 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:07.802+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:07.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:07.899 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:07.897+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:07.957 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:07.957 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:07.957 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:07.957 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:07.957 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:07.965 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:08.009 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:08.008+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:08.070 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:08.070 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.318530483000004, 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19491629299999999, 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.071 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748958+0000", 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748966+0000", 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5870000000000004e-06 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748979+0000", 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-09-17T16:02:08.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748988+0000", 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749052+0000", 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4618999999999999e-05 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:08.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749064+0000", 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.939947+0000", 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19088328399999999 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940079+0000", 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013157500000000001 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940245+0000", 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016577900000000001 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943345+0000", 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031004499999999998 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:08.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943806+0000", 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046112200000000001 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943843+0000", 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6578000000000001e-05 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943873+0000", 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.077 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [cmpxattr striper.excl (25) op 1 mode 1 in=12b,setxattr striper.excl in=12b,call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.119445157000001, 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83632086699999997, 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.078 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948043+0000", 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948048+0000", 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948060+0000", 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1761e-05 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948066+0000", 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5350000000000001e-06 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948111+0000", 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4783000000000001e-05 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948190+0000", 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948399+0000", 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000209292 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948922+0000", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052251800000000003 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784264+0000", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83534246599999995 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784309+0000", 2024-09-17T16:02:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4660000000000003e-05 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784363+0000", 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4095000000000003e-05 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.4219.0:8 1.0 e24/23 ondisk, result = 0)", 2024-09-17T16:02:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.118322962000001, 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83521008600000002, 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949165+0000", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949169+0000", 2024-09-17T16:02:08.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9339999999999999e-06 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784135+0000", 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83496651200000005 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784151+0000", 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784250+0000", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8222999999999998e-05 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784260+0000", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0265000000000001e-05 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784374+0000", 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114373 2024-09-17T16:02:08.088 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4296.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 49.308134748999997, 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.241196405, 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.089 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4296", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.6:0/1428345084", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759356+0000", 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.484e-06 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759366+0000", 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9089999999999995e-06 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759375+0000", 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759397+0000", 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1824e-05 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759444+0000", 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6539000000000003e-05 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759449+0000", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998167+0000", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23871730099999999 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998242+0000", 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4892000000000001e-05 2024-09-17T16:02:08.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998303+0000", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0931999999999999e-05 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998708+0000", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040511399999999998 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000499+0000", 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017911909999999999 2024-09-17T16:02:08.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000526+0000", 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6798999999999999e-05 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000549+0000", 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3110000000000001e-05 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:34 3.2s0 3:41aa4422:::smithi18136759-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.210872244999997, 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015312129000000001, 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 34 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.096 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856616+0000", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4899999999999999e-06 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856732+0000", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011513 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856745+0000", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3677e-05 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856753+0000", 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856824+0000", 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1291999999999994e-05 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856962+0000", 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013730799999999999 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.857708+0000", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074624700000000001 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861225+0000", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035171249999999999 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871848+0000", 2024-09-17T16:02:08.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010623323 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871927+0000", 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8844000000000006e-05 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:35 3.2s0 3:41aa4422:::smithi18136759-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.210716705000003, 2024-09-17T16:02:08.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015326866, 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 35 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856771+0000", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856775+0000", 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.33e-06 2024-09-17T16:02:08.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856782+0000", 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0800000000000002e-06 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856786+0000", 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5759999999999999e-06 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858308+0000", 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015218009999999999 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858393+0000", 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5502999999999998e-05 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859025+0000", 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000632009 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861267+0000", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022417639999999998 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872026+0000", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010758772999999999 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872097+0000", 2024-09-17T16:02:08.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1390000000000006e-05 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:36 3.2s0 3:41aa4422:::smithi18136759-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.210631362000001, 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017199677, 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.107 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 36 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856861+0000", 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0759999999999996e-06 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856869+0000", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6319999999999997e-06 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856876+0000", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5989999999999998e-06 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859529+0000", 2024-09-17T16:02:08.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026531060000000001 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859628+0000", 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9104000000000005e-05 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872415+0000", 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012787056 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872500+0000", 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999998e-05 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874055+0000", 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015554810000000001 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.112 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=11, reqid=client.4560.0:50, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.196547697, 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016547899000000001, 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:08.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870940+0000", 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870953+0000", 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2798e-05 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881713+0000", 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010760202 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881721+0000", 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8949999999999998e-06 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881765+0000", 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3778e-05 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881775+0000", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0135999999999999e-05 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887451+0000", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056764090000000003 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887487+0000", 2024-09-17T16:02:08.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6065000000000001e-05 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs2 35/32 ECSubWrite(tid=20, reqid=client.4560.0:74, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.891717+0000", 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.175769913000003, 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014297703, 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.117 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.891717+0000", 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.891717+0000", 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.891718+0000", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2900000000000003e-07 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.891737+0000", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8972999999999999e-05 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905202+0000", 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.013464689 2024-09-17T16:02:08.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905210+0000", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3219999999999993e-06 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905267+0000", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7256999999999997e-05 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905276+0000", 2024-09-17T16:02:08.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8599999999999999e-06 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.905979+0000", 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070289400000000002 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.906015+0000", 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6078999999999998e-05 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.121 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:81 3.2s0 3:4c2fa52d:::smithi18136759-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.173443898000002, 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015999928, 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 81 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894044+0000", 2024-09-17T16:02:08.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894045+0000", 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065e-06 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894048+0000", 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.489e-06 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894051+0000", 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9799999999999998e-06 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896400+0000", 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023486919999999999 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896473+0000", 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3380000000000006e-05 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908531+0000", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012058016 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908615+0000", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3968000000000007e-05 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.910043+0000", 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014280569999999999 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s3 35/32 ECSubWrite(tid=34, reqid=client.4560.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.142088047000001, 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021043124999999999, 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:08.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925400+0000", 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925410+0000", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0345e-05 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944975+0000", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019565013999999999 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944985+0000", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0257e-05 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945787+0000", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00080139199999999995 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945797+0000", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9590000000000007e-06 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946410+0000", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061359000000000001 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946442+0000", 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=42, reqid=client.4560.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.111730023, 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022376667999999999, 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955758+0000", 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955765+0000", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.345e-06 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977438+0000", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021672416999999999 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977447+0000", 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977511+0000", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4066000000000006e-05 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977519+0000", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8329999999999994e-06 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978105+0000", 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058612099999999999 2024-09-17T16:02:08.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978134+0000", 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8558e-05 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:268 3.2s0 3:406bd9c9:::smithi18136759-23:head [append 0~720896] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.144163518999999, 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016657371000000001, 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.135 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 268 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923326+0000", 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0999999999999998e-06 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923806+0000", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048034 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923821+0000", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5180999999999999e-05 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923831+0000", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4450000000000002e-06 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923891+0000", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9858999999999997e-05 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923971+0000", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0493000000000003e-05 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.924982+0000", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001010543 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929374+0000", 2024-09-17T16:02:08.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043921699999999999 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939921+0000", 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010546754 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939981+0000", 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0486000000000003e-05 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:269 3.2s0 3:406bd9c9:::smithi18136759-23:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925517+0000", 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.141970529999998, 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014617791999999999, 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 269 2024-09-17T16:02:08.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925517+0000", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925517+0000", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925518+0000", 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5409999999999999e-06 2024-09-17T16:02:08.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925867+0000", 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034877500000000001 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925882+0000", 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4605e-05 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925891+0000", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1309999999999995e-06 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926292+0000", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040123999999999997 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.926431+0000", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138963 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927209+0000", 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077788100000000002 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032671229999999998 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940074+0000", 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0095984759999999999 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940134+0000", 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0056999999999998e-05 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:270 3.2s0 3:406bd9c9:::smithi18136759-23:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925911+0000", 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.141576369999999, 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014341911000000001, 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.146 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 270 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925911+0000", 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925911+0000", 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925912+0000", 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8199999999999998e-07 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925916+0000", 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7709999999999997e-06 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925926+0000", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5270000000000004e-06 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925932+0000", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0909999999999998e-06 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928031+0000", 2024-09-17T16:02:08.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0020987520000000002 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928133+0000", 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102608 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928650+0000", 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00051676800000000002 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.930479+0000", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001828477 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940204+0000", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0097254200000000002 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940253+0000", 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8615000000000001e-05 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.151 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:271 3.2s0 3:406bd9c9:::smithi18136759-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 24.141527450000002, 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015374301, 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.152 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 271 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925961+0000", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925964+0000", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5319999999999998e-06 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925976+0000", 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1795e-05 2024-09-17T16:02:08.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925983+0000", 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.289e-06 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929232+0000", 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032491070000000002 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929352+0000", 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119639 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940555+0000", 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011203489000000001 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940634+0000", 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8165999999999997e-05 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.941334+0000", 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000700496 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:392 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~589824] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7250108199999996, 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018219264999999998, 2024-09-17T16:02:08.157 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 392 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342479+0000", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.481e-06 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343310+0000", 2024-09-17T16:02:08.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083096699999999995 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343328+0000", 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8059000000000001e-05 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343340+0000", 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1664e-05 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343445+0000", 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010582900000000001 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343453+0000", 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348803+0000", 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053503420000000001 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348975+0000", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017183899999999999 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350509+0000", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015333320000000001 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356400+0000", 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058917320000000002 2024-09-17T16:02:08.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360612+0000", 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042120680000000002 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360696+0000", 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:393 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~573440] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7212537920000006, 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014645886, 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 393 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346236+0000", 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2409999999999998e-06 2024-09-17T16:02:08.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346934+0000", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069824199999999996 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346952+0000", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7728999999999999e-05 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346961+0000", 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0159999999999997e-06 2024-09-17T16:02:08.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347031+0000", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0168999999999999e-05 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347039+0000", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2190000000000004e-06 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.351923+0000", 2024-09-17T16:02:08.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048838550000000003 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352046+0000", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012300599999999999 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352744+0000", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069777000000000001 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356401+0000", 2024-09-17T16:02:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036577049999999998 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360814+0000", 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044128459999999998 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360879+0000", 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5087999999999998e-05 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:396 3.2s0 3:465e1f11:::smithi18136759-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 9.7197494140000007, 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015698437999999999, 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 396 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8699999999999995e-07 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347741+0000", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6690000000000002e-06 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347747+0000", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.455e-06 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347752+0000", 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0459999999999996e-06 2024-09-17T16:02:08.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356238+0000", 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084855650000000005 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356325+0000", 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7039999999999993e-05 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361673+0000", 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053483849999999998 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361762+0000", 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8825000000000004e-05 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.363436+0000", 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001673966 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.174 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:484 3.18s0 3:19aefe3f:::smithi18136759-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.71652931399999997, 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017624631000000002, 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 484 2024-09-17T16:02:08.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350961+0000", 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352262+0000", 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001300708 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352278+0000", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5848000000000001e-05 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354256+0000", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019783330000000001 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354327+0000", 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0839000000000004e-05 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363568+0000", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092410559999999992 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363668+0000", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9382000000000005e-05 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368582+0000", 2024-09-17T16:02:08.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049148810000000003 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.180 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:08.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:08.182 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:08.180+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:08.199 INFO:tasks.rados.rados.0.smithi181.stdout:196: finishing write tid 4 to smithi18136759-45 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: finishing write tid 5 to smithi18136759-45 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: oid 45 updating version 0 to 40 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: oid 45 version 40 is already newer than 39 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: oid 45 updating version 40 to 42 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: oid 45 version 42 is already newer than 41 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 45 v 42 (ObjNum 90 snap 22 seq_num 90) dirty exists 2024-09-17T16:02:08.201 INFO:tasks.rados.rados.0.smithi181.stdout:196: left oid 45 (ObjNum 90 snap 22 seq_num 90) 2024-09-17T16:02:08.282 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:08.279+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:08.301 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:08.301 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:08.301 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:08.301 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:08.309 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:08.365 INFO:tasks.rados.rados.0.smithi181.stdout:196: done (2 left) 2024-09-17T16:02:08.365 INFO:tasks.rados.rados.0.smithi181.stdout:202: done (1 left) 2024-09-17T16:02:08.365 INFO:tasks.rados.rados.0.smithi181.stdout:204: done (0 left) 2024-09-17T16:02:08.365 INFO:tasks.rados.rados.0.smithi181.stdout:205: setattr oid 48 current snap is 23 2024-09-17T16:02:08.366 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 18 current snap is 23 2024-09-17T16:02:08.366 INFO:tasks.rados.rados.0.smithi181.stdout:206: seq_num 94 ranges {0=425984,425984=786432,1212416=376832} 2024-09-17T16:02:08.368 INFO:tasks.rados.rados.0.smithi181.stdout:206: writing smithi18136759-18 from 0 to 425984 tid 1 2024-09-17T16:02:08.373 INFO:tasks.rados.rados.0.smithi181.stdout:206: writing smithi18136759-18 from 425984 to 1212416 tid 2 2024-09-17T16:02:08.376 INFO:tasks.rados.rados.0.smithi181.stdout:206: writing smithi18136759-18 from 1212416 to 1589248 tid 3 2024-09-17T16:02:08.376 INFO:tasks.rados.rados.0.smithi181.stdout:207: copy_from oid 22 from oid 26 current snap is 23 2024-09-17T16:02:08.376 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 52 (ObjNum 49 snap 0 seq_num 1953066355) dirty exists 2024-09-17T16:02:08.376 INFO:tasks.rados.rados.0.smithi181.stdout:205: done (2 left) 2024-09-17T16:02:08.376 INFO:tasks.rados.rados.0.smithi181.stdout:208: delete oid 28 current snap is 23 2024-09-17T16:02:08.379 INFO:tasks.rados.rados.0.smithi181.stdout:206: finishing write tid 1 to smithi18136759-18 2024-09-17T16:02:08.382 INFO:tasks.rados.rados.0.smithi181.stdout:208: done (2 left) 2024-09-17T16:02:08.382 INFO:tasks.rados.rados.0.smithi181.stdout:209: copy_from oid 49 from oid 2 current snap is 23 2024-09-17T16:02:08.382 INFO:tasks.rados.rados.0.smithi181.stdout:210: snap_create 2024-09-17T16:02:08.383 INFO:tasks.rados.rados.0.smithi181.stdout:206: finishing write tid 2 to smithi18136759-18 2024-09-17T16:02:08.383 INFO:tasks.rados.rados.0.smithi181.stdout:206: finishing write tid 3 to smithi18136759-18 2024-09-17T16:02:08.383 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:08.380+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:08.387 INFO:tasks.rados.rados.0.smithi181.stdout:207: finishing copy_from racing read to smithi18136759-22 2024-09-17T16:02:08.391 INFO:tasks.rados.rados.0.smithi181.stdout:207: finishing copy_from to smithi18136759-22 2024-09-17T16:02:08.391 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T16:02:08.397 INFO:tasks.rados.rados.0.smithi181.stdout:209: finishing copy_from racing read to smithi18136759-49 2024-09-17T16:02:08.398 INFO:tasks.rados.rados.0.smithi181.stdout:209: finishing copy_from to smithi18136759-49 2024-09-17T16:02:08.398 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 49 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-17T16:02:08.417 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:08.417 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:08.417 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:08.417 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:08.417 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:08.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:08.483 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:08.481+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:08.530 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:08.530 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:08.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:08.530 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:08.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.667534383000003, 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053801109999999999, 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860418+0000", 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020234499999999999 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860448+0000", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0692999999999999e-05 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860458+0000", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6460000000000005e-06 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860528+0000", 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0272999999999996e-05 2024-09-17T16:02:08.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860536+0000", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1040000000000007e-06 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865550+0000", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050133510000000001 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865595+0000", 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5396999999999999e-05 2024-09-17T16:02:08.534 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.663498658999998, 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051579750000000004, 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864260+0000", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5769999999999994e-06 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864296+0000", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6227999999999999e-05 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864304+0000", 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0090000000000007e-06 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864392+0000", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8549999999999998e-05 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864402+0000", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3279999999999995e-06 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869377+0000", 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004975045 2024-09-17T16:02:08.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869409+0000", 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1953999999999999e-05 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.660286243999998, 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.539 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:02:08.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:02:08.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.659896531999998, 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:08.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:02:08.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:02:08.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.650476122999997, 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.552 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595790598000001, 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.553 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:08.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.558 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595408198999998, 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:02:08.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.564 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595372406000003, 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:08.565 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.566 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.567 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.568 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.569 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595344240999999, 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.570 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.571 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:02:08.572 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.573 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:02:08.574 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:08.575 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595070722000003, 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:08.576 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.577 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:02:08.578 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:02:08.579 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.580 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.595059665999997, 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:02:08.581 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.582 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:02:08.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:02:08.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.562304513000001, 2024-09-17T16:02:08.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.591 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.561815463000002, 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.592 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:08.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:02:08.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.597 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.561774612999997, 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.598 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:02:08.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:02:08.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.614055016999998, 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073441670000000004, 2024-09-17T16:02:08.603 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913695+0000", 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913725+0000", 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9743999999999998e-05 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919758+0000", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00603318 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919766+0000", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3610000000000001e-06 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919835+0000", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9336000000000002e-05 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919844+0000", 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921001+0000", 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115653 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921039+0000", 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7700000000000002e-05 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 46/32 ECSubWriteReply(tid=17, last_complete=46'12, committed=1, applied=1))", 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.604068439999999, 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.330502611, 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:08.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923682+0000", 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923686+0000", 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253962+0000", 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33027621400000001 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253975+0000", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254119+0000", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014402100000000001 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254184+0000", 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4158999999999993e-05 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.611 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:346 3.3s0 3:c1daf1ed:::smithi18136759-38:head [delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.168979619000002, 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069854690000000002, 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.612 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358772+0000", 2024-09-17T16:02:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0810000000000001e-06 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358779+0000", 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2849999999999997e-06 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358790+0000", 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9759999999999995e-06 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358838+0000", 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9730000000000001e-05 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358966+0000", 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012749199999999999 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.359434+0000", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046890000000000001 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.362289+0000", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028544030000000001 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365699+0000", 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034100210000000001 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365755+0000", 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6456000000000003e-05 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.10s3 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1816081790000004, 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081444710000000003, 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.617 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:08.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-07 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346159+0000", 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7445e-05 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346307+0000", 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014812199999999999 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346314+0000", 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.477e-06 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346436+0000", 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012286999999999999 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346445+0000", 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.833e-06 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354242+0000", 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077967399999999999 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354286+0000", 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3801999999999998e-05 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.16s2 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.1792763580000001, 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059176979999999999, 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348474+0000", 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5799999999999999e-07 2024-09-17T16:02:08.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348485+0000", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0892e-05 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348510+0000", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5287999999999999e-05 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348517+0000", 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.267e-06 2024-09-17T16:02:08.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348561+0000", 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4360000000000002e-05 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348568+0000", 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7270000000000001e-06 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354355+0000", 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057869480000000001 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354391+0000", 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5658000000000003e-05 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.18s3 69/53 ECSubWrite(tid=217, reqid=client.4560.0:481, at_version=69'35, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:08.626 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.177367566, 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050637679999999997, 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:08.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350383+0000", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350488+0000", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010493999999999999 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350525+0000", 2024-09-17T16:02:08.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7608000000000003e-05 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350534+0000", 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350610+0000", 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6055000000000003e-05 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350621+0000", 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0485999999999999e-05 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.355408+0000", 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047872100000000001 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.355446+0000", 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7858999999999998e-05 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.630 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.631 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:08.631 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:08.631 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:08.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:08.633 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:08.629+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:08.733 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:08.729+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:08.752 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:08.752 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:08.752 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:08.753 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:08.762 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:08.833 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:08.831+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:08.869 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:08.869 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:08.870 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:08.870 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:08.870 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:08.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:08.934 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:08.932+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:235 3.fs0 3:fe5cbd82:::smithi18136759-11:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.062771730000001, 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072733650000000004, 2024-09-17T16:02:08.987 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:08.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921352+0000", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8500000000000002e-07 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921355+0000", 2024-09-17T16:02:08.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.084e-06 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921360+0000", 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5079999999999999e-06 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.921363+0000", 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1109999999999999e-06 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:08.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922189+0000", 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082667099999999996 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922288+0000", 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8857999999999994e-05 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.922688+0000", 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00039985699999999999 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.991 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.927817+0000", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005129013 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928567+0000", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00075011800000000003 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928625+0000", 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7760000000000003e-05 2024-09-17T16:02:08.992 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:248 3.cs0 3:3808462d:::smithi18136759-6:head [copy-from ver 2] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.055541334000001, 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019268496999999999, 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:08.993 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 248 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:08.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928583+0000", 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928588+0000", 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2220000000000001e-06 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:08.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928601+0000", 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2717e-05 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928608+0000", 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8600000000000004e-06 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:08.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928671+0000", 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3791e-05 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928756+0000", 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4293000000000001e-05 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940817+0000", 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01206144 2024-09-17T16:02:08.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.942933+0000", 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021155280000000002 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947791+0000", 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048580639999999996 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:08.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947850+0000", 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9682999999999998e-05 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:249 3.cs0 3:3808462d:::smithi18136759-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 26.055500221999999, 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013329483, 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:08.999 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928624+0000", 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2699999999999999e-07 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928626+0000", 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8940000000000002e-06 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928643+0000", 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6988999999999999e-05 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928651+0000", 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.267e-06 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928901+0000", 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025058200000000001 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928955+0000", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3973e-05 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941770+0000", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012814828 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941852+0000", 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083000000000007e-05 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941953+0000", 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010034 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:285 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.052317902999999, 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010247877000000001, 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.004 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 285 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931808+0000", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8779999999999998e-06 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931822+0000", 2024-09-17T16:02:09.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3828e-05 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931850+0000", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7500000000000001e-05 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931863+0000", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3738e-05 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931969+0000", 2024-09-17T16:02:09.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001052 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931976+0000", 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938082+0000", 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061057769999999997 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938176+0000", 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3714000000000004e-05 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939069+0000", 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00089279400000000003 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941333+0000", 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022647380000000001 2024-09-17T16:02:09.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941992+0000", 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065853199999999998 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.942053+0000", 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1447999999999998e-05 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.010 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:286 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 2] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.052164213000001, 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013358431, 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 286 2024-09-17T16:02:09.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931960+0000", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931965+0000", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9010000000000002e-06 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931974+0000", 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2520000000000005e-06 2024-09-17T16:02:09.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931981+0000", 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5950000000000004e-06 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932071+0000", 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0390999999999996e-05 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:09.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932079+0000", 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2679999999999994e-06 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939566+0000", 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074865030000000003 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939653+0000", 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7336999999999994e-05 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940238+0000", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058448400000000002 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944090+0000", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00385192 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945232+0000", 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001142784 2024-09-17T16:02:09.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945317+0000", 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5080999999999994e-05 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:287 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.052128274000001, 2024-09-17T16:02:09.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013571456000000001, 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 287 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931996+0000", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931999+0000", 2024-09-17T16:02:09.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4300000000000002e-06 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932006+0000", 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2729999999999997e-06 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932012+0000", 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8089999999999998e-06 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932107+0000", 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4751000000000001e-05 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932112+0000", 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940481+0000", 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083682089999999997 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940546+0000", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5704000000000007e-05 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940923+0000", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00037686099999999998 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944112+0000", 2024-09-17T16:02:09.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031884999999999999 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945503+0000", 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001391226 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945566+0000", 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3510999999999996e-05 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 52/32 ECSubWriteReply(tid=51, last_complete=52'22, committed=1, applied=1))", 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.627152121000002, 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.138524809, 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356972+0000", 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:09.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356976+0000", 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2950000000000004e-06 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495052+0000", 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13807575 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495065+0000", 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3276e-05 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495133+0000", 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8274000000000004e-05 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495496+0000", 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036245899999999998 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:344 3.fs0 3:fd9ed431:::smithi18136759-28:head [stat,delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.648160191999999, 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012105121, 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 344 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.028 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335966+0000", 2024-09-17T16:02:09.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.004e-06 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335978+0000", 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1810999999999999e-05 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336003+0000", 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5344e-05 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336017+0000", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3901e-05 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336125+0000", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107707 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336136+0000", 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1476e-05 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340410+0000", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004273453 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340525+0000", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115017 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.341450+0000", 2024-09-17T16:02:09.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00092533800000000005 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.345523+0000", 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040730929999999999 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348008+0000", 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024844609999999999 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348068+0000", 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0516000000000001e-05 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=128, reqid=osd.12.0:128, at_version=64'68, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6462250210000002, 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076656809999999997, 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.034 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337917+0000", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8366999999999999e-05 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339622+0000", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017049529999999999 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339629+0000", 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3660000000000003e-06 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339905+0000", 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027553099999999997 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339915+0000", 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0124e-05 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345540+0000", 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056253780000000003 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799e-05 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.038 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6444665350000003, 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081855320000000006, 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339692+0000", 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4459000000000003e-05 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339726+0000", 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4029000000000002e-05 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339733+0000", 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0790000000000003e-06 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340271+0000", 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00053823 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340282+0000", 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347801+0000", 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075190999999999999 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347842+0000", 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1387999999999997e-05 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.10s6 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6385596429999998, 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011676772, 2024-09-17T16:02:09.043 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.29e-07 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8580000000000001e-05 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346013+0000", 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00042032499999999998 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346021+0000", 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2269999999999993e-06 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346353+0000", 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033160900000000002 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346363+0000", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0390999999999999e-05 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357206+0000", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010843148 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357240+0000", 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4263e-05 2024-09-17T16:02:09.047 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6376525900000001, 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010663848, 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.048 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7099999999999998e-07 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346489+0000", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7731999999999999e-05 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346664+0000", 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017518199999999999 2024-09-17T16:02:09.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346671+0000", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1239999999999999e-06 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346721+0000", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.02e-05 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.55e-06 2024-09-17T16:02:09.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357095+0000", 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010364226000000001 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357135+0000", 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9563000000000001e-05 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s5 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6362729140000001, 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092773109999999999, 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347851+0000", 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347861+0000", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0325e-05 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347893+0000", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1173000000000001e-05 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347900+0000", 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.306e-06 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347964+0000", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3678999999999998e-05 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347973+0000", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0790000000000007e-06 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357093+0000", 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091199190000000006 2024-09-17T16:02:09.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357128+0000", 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5055999999999998e-05 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6349971740000004, 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090524279999999995, 2024-09-17T16:02:09.057 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349127+0000", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349137+0000", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349186+0000", 2024-09-17T16:02:09.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8306999999999997e-05 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349195+0000", 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3640000000000005e-06 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349414+0000", 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021941500000000001 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349424+0000", 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358144+0000", 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087197060000000007 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358179+0000", 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4814999999999998e-05 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=132, reqid=osd.12.0:132, at_version=64'72, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6325422630000004, 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076285509999999999, 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.062 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351582+0000", 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351615+0000", 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2962000000000001e-05 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351654+0000", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8847999999999998e-05 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351662+0000", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0099999999999995e-06 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351779+0000", 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117569 2024-09-17T16:02:09.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351788+0000", 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5639999999999997e-06 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359157+0000", 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073690580000000004 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359210+0000", 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2769999999999998e-05 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.6316767810000004, 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079374270000000004, 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.067 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352447+0000", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352467+0000", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0146999999999999e-05 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352504+0000", 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6365000000000002e-05 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352512+0000", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352565+0000", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3613999999999999e-05 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352576+0000", 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1141999999999999e-05 2024-09-17T16:02:09.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360356+0000", 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007779584 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360384+0000", 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7847999999999998e-05 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.6307023410000001, 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70314915600000005, 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.54e-07 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353470+0000", 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9177e-05 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055585+0000", 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70211433599999995 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055600+0000", 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4989e-05 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055790+0000", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019022599999999999 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055803+0000", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3455000000000001e-05 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056540+0000", 2024-09-17T16:02:09.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000736289 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056570+0000", 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0530000000000001e-05 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:490 3.fs0 3:fd9ed431:::smithi18136759-28:head [append 0~770048] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:09.076 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.6151046019999999, 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007518299, 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 490 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:09.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369020+0000", 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.322e-06 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369811+0000", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00079078099999999995 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369823+0000", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.252e-05 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369829+0000", 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4140000000000002e-06 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369879+0000", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0158999999999999e-05 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369990+0000", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111549 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.370711+0000", 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00072055199999999995 2024-09-17T16:02:09.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.371929+0000", 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001218217 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376334+0000", 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044050360000000002 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376537+0000", 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020274900000000001 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:492 3.fs0 3:fd9ed431:::smithi18136759-28:head [append 0~294912] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.373356+0000", 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.610767563, 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073194449999999999, 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:09.082 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 492 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373356+0000", 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373356+0000", 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373356+0000", 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5000000000000003e-07 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373613+0000", 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025677300000000002 2024-09-17T16:02:09.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373621+0000", 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3149999999999992e-06 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373630+0000", 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8880000000000003e-06 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373674+0000", 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3733999999999998e-05 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373787+0000", 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000112645 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.374455+0000", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00066871400000000005 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376195+0000", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001739747 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.380575+0000", 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043797180000000003 2024-09-17T16:02:09.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.380675+0000", 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100361 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:494 3.fs0 3:fd9ed431:::smithi18136759-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:09.088 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 1.610453744, 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082749039999999996, 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 494 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:09.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.51e-07 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373672+0000", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2689999999999998e-06 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373678+0000", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2909999999999998e-06 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373681+0000", 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3730000000000001e-06 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376047+0000", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002366071 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376106+0000", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8610000000000003e-05 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381155+0000", 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050492460000000003 2024-09-17T16:02:09.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381250+0000", 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5358000000000004e-05 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381944+0000", 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069423499999999997 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:09.093 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:09.094 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:09.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:09.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:09.096 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:09.093+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:09.196 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:09.193+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:09.245 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:09.246 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:09.246 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:09.246 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:09.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:09.272 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:09.272 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:09.294 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:09.294 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:09.295 INFO:tasks.rados.rados.0.smithi181.stdout:206: finishing write tid 4 to smithi18136759-18 2024-09-17T16:02:09.296 INFO:tasks.rados.rados.0.smithi181.stdout:206: finishing write tid 5 to smithi18136759-18 2024-09-17T16:02:09.296 INFO:tasks.rados.rados.0.smithi181.stdout:206: oid 18 updating version 0 to 102 2024-09-17T16:02:09.297 INFO:tasks.rados.rados.0.smithi181.stdout:206: oid 18 updating version 102 to 105 2024-09-17T16:02:09.297 INFO:tasks.rados.rados.0.smithi181.stdout:206: oid 18 version 105 is already newer than 104 2024-09-17T16:02:09.297 INFO:tasks.rados.rados.0.smithi181.stdout:206: oid 18 version 105 is already newer than 103 2024-09-17T16:02:09.297 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 105 (ObjNum 94 snap 23 seq_num 94) dirty exists 2024-09-17T16:02:09.297 INFO:tasks.rados.rados.0.smithi181.stdout:206: left oid 18 (ObjNum 94 snap 23 seq_num 94) 2024-09-17T16:02:09.297 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:09.295+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:09.316 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:09.317 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:09.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:09.338 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:09.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:02:09.359 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:09.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:09.364 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:09.364 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:09.370 INFO:tasks.rados.rados.0.smithi181.stdout:206: done (3 left) 2024-09-17T16:02:09.371 INFO:tasks.rados.rados.0.smithi181.stdout:207: done (2 left) 2024-09-17T16:02:09.371 INFO:tasks.rados.rados.0.smithi181.stdout:209: done (1 left) 2024-09-17T16:02:09.371 INFO:tasks.rados.rados.0.smithi181.stdout:210: done (0 left) 2024-09-17T16:02:09.371 INFO:tasks.rados.rados.0.smithi181.stdout:211: snap_remove snap 6 2024-09-17T16:02:09.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:09.387 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:09.388 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:09.409 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:09.409 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:09.410 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:09.407+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:09.410 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:09.432 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:09.432 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:09.453 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:09.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:09.475 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:09.476 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:09.498 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:09.496+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:09.499 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:09.500 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:09.500 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:09.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:09.522 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:09.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:09.559 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:09.559 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:09.580 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:09.579+0000 7f01dffff640 0 -- 172.21.15.6:0/1605351331 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f01e821fab0 msgr2=0x7f01e8221ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:09.599 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:09.596+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:09.630 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:09.630 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:09.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:09.699 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:09.698+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:09.786 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:09.786 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:09.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:09.800 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:09.799+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:09.803 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:09.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:09.902 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:09.900+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:09.903 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:09.903 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:09.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:10.002 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:10.001+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:10.059 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.059 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:10.104 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:10.101+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:10.174 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.175 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:10.205 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:10.203+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:10.289 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.290 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:10.305 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:10.303+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:10.368 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:02:10.372 INFO:tasks.rados.rados.0.smithi181.stdout:211: done (0 left) 2024-09-17T16:02:10.372 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 8 current snap is 24 2024-09-17T16:02:10.373 INFO:tasks.rados.rados.0.smithi181.stdout:212: seq_num 96 ranges {0=458752,458752=688128,1146880=475136} 2024-09-17T16:02:10.375 INFO:tasks.rados.rados.0.smithi181.stdout:212: writing smithi18136759-8 from 0 to 458752 tid 1 2024-09-17T16:02:10.379 INFO:tasks.rados.rados.0.smithi181.stdout:212: writing smithi18136759-8 from 458752 to 1146880 tid 2 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:212: writing smithi18136759-8 from 1146880 to 1622016 tid 3 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:213: copy_from oid 3 from oid 6 current snap is 24 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:214: read oid 9 snap -1 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:214: expect deleted 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:215: rollback oid 5 current snap is 24 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 5 to 12 2024-09-17T16:02:10.383 INFO:tasks.rados.rados.0.smithi181.stdout:216: delete oid 43 current snap is 24 2024-09-17T16:02:10.390 INFO:tasks.rados.rados.0.smithi181.stdout:212: finishing write tid 1 to smithi18136759-8 2024-09-17T16:02:10.396 INFO:tasks.rados.rados.0.smithi181.stdout:216: done (4 left) 2024-09-17T16:02:10.396 INFO:tasks.rados.rados.0.smithi181.stdout:217: copy_from oid 13 from oid 36 current snap is 24 2024-09-17T16:02:10.396 INFO:tasks.rados.rados.0.smithi181.stdout:218: rollback oid 4 current snap is 24 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 4 to 18 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:219: read oid 17 snap 12 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:219: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:220: rollback oid 48 current snap is 24 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 48 to 11 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 24 2024-09-17T16:02:10.397 INFO:tasks.rados.rados.0.smithi181.stdout:221: seq_num 98 ranges {311296=491520,802816=589824,1392640=557056,1949696=16384} 2024-09-17T16:02:10.401 INFO:tasks.rados.rados.0.smithi181.stdout:221: writing smithi18136759-31 from 311296 to 802816 tid 1 2024-09-17T16:02:10.404 INFO:tasks.rados.rados.0.smithi181.stdout:221: writing smithi18136759-31 from 802816 to 1392640 tid 2 2024-09-17T16:02:10.405 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:10.404+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:10.408 INFO:tasks.rados.rados.0.smithi181.stdout:221: writing smithi18136759-31 from 1392640 to 1949696 tid 3 2024-09-17T16:02:10.408 INFO:tasks.rados.rados.0.smithi181.stdout:221: writing smithi18136759-31 from 1949696 to 1966080 tid 4 2024-09-17T16:02:10.408 INFO:tasks.rados.rados.0.smithi181.stdout:222: delete oid 16 current snap is 24 2024-09-17T16:02:10.408 INFO:tasks.rados.rados.0.smithi181.stdout:212: finishing write tid 2 to smithi18136759-8 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:215: finishing rollback tid 0 to smithi18136759-5 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: finishing write tid 3 to smithi18136759-8 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: finishing write tid 4 to smithi18136759-8 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:215: finishing rollback tid 1 to smithi18136759-5 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:215: finishing rollback tid 2 to smithi18136759-5 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 5 v 41 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: finishing write tid 5 to smithi18136759-8 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: oid 8 updating version 0 to 106 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: oid 8 updating version 106 to 111 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: oid 8 updating version 111 to 113 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: oid 8 version 113 is already newer than 112 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 113 (ObjNum 96 snap 24 seq_num 96) dirty exists 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:212: left oid 8 (ObjNum 96 snap 24 seq_num 96) 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:219: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-17T16:02:10.409 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:10.411 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:10.414 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing copy_from to smithi18136759-13 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:217: got expected ENOENT (src dne) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:217: finishing copy_from racing read to smithi18136759-13 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:212: done (9 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:214: done (8 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:215: done (7 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:217: done (6 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:219: done (5 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:222: done (4 left) 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:223: read oid 38 snap -1 2024-09-17T16:02:10.419 INFO:tasks.rados.rados.0.smithi181.stdout:223: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:224: copy_from oid 13 from oid 33 current snap is 24 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:218: finishing rollback tid 0 to smithi18136759-4 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:218: finishing rollback tid 1 to smithi18136759-4 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:220: finishing rollback tid 0 to smithi18136759-48 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:220: finishing rollback tid 1 to smithi18136759-48 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:218: finishing rollback tid 2 to smithi18136759-4 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 58 (ObjNum 33 snap 0 seq_num 1454770448) dirty exists 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:220: finishing rollback tid 2 to smithi18136759-48 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 60 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 1 to smithi18136759-31 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 2 to smithi18136759-31 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 3 to smithi18136759-31 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 4 to smithi18136759-31 2024-09-17T16:02:10.420 INFO:tasks.rados.rados.0.smithi181.stdout:218: done (5 left) 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:220: done (4 left) 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:225: copy_from oid 40 from oid 30 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:226: rmattr oid 46 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:226: done (5 left) 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:227: rollback oid 8 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 8 to 22 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:228: copy_from oid 14 from oid 48 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:229: rmattr oid 46 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:229: done (7 left) 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:230: rollback oid 23 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 23 to 11 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 27 current snap is 24 2024-09-17T16:02:10.421 INFO:tasks.rados.rados.0.smithi181.stdout:231: seq_num 100 ranges {442368=524288,966656=753664,1720320=409600} 2024-09-17T16:02:10.423 INFO:tasks.rados.rados.0.smithi181.stdout:231: writing smithi18136759-27 from 442368 to 966656 tid 1 2024-09-17T16:02:10.428 INFO:tasks.rados.rados.0.smithi181.stdout:231: writing smithi18136759-27 from 966656 to 1720320 tid 2 2024-09-17T16:02:10.430 INFO:tasks.rados.rados.0.smithi181.stdout:231: writing smithi18136759-27 from 1720320 to 2129920 tid 3 2024-09-17T16:02:10.432 INFO:tasks.rados.rados.0.smithi181.stdout:232: rollback oid 35 current snap is 24 2024-09-17T16:02:10.432 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 35 to 4 2024-09-17T16:02:10.432 INFO:tasks.rados.rados.0.smithi181.stdout:223: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 5 to smithi18136759-31 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:230: finishing rollback tid 0 to smithi18136759-23 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:230: finishing rollback tid 1 to smithi18136759-23 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:227: finishing rollback tid 0 to smithi18136759-8 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:223: done (9 left) 2024-09-17T16:02:10.433 INFO:tasks.rados.rados.0.smithi181.stdout:233: snap_remove snap 18 2024-09-17T16:02:10.444 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.444 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:10.506 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:10.504+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:10.562 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.562 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:10.607 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:10.605+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:10.677 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.677 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:10.708 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:10.706+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:10.792 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.792 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:10.844 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:10.841+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:10.907 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:10.908 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:10.910 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:10.908+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:10.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:11.011 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:11.008+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:11.063 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.063 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:11.112 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:11.111+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:11.213 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:11.211+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:11.219 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.219 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:11.313 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:11.311+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:11.333 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.334 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:233: done (9 left) 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:234: read oid 18 snap -1 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:234: expect (ObjNum 94 snap 23 seq_num 94) 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:235: delete oid 20 current snap is 24 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:221: finishing write tid 6 to smithi18136759-31 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:221: oid 31 updating version 0 to 39 2024-09-17T16:02:11.384 INFO:tasks.rados.rados.0.smithi181.stdout:221: oid 31 updating version 39 to 41 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:221: oid 31 version 41 is already newer than 40 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:221: oid 31 version 41 is already newer than 37 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:221: oid 31 version 41 is already newer than 38 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 41 (ObjNum 98 snap 24 seq_num 98) dirty exists 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:221: left oid 31 (ObjNum 98 snap 24 seq_num 98) 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:227: finishing rollback tid 1 to smithi18136759-8 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 117 (ObjNum 33 snap 0 seq_num 1475200224) dirty dne 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:230: finishing rollback tid 2 to smithi18136759-23 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 23 v 45 (ObjNum 65 snap 9 seq_num 65) dirty exists 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:231: finishing write tid 1 to smithi18136759-27 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:231: finishing write tid 2 to smithi18136759-27 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:232: finishing rollback tid 0 to smithi18136759-35 2024-09-17T16:02:11.385 INFO:tasks.rados.rados.0.smithi181.stdout:231: finishing write tid 3 to smithi18136759-27 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing copy_from racing read to smithi18136759-13 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: finishing write tid 4 to smithi18136759-27 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:232: finishing rollback tid 1 to smithi18136759-35 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:232: finishing rollback tid 2 to smithi18136759-35 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 19 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:228: finishing copy_from racing read to smithi18136759-14 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: finishing write tid 5 to smithi18136759-27 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: oid 27 updating version 0 to 36 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: oid 27 updating version 36 to 38 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: oid 27 version 38 is already newer than 37 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: oid 27 updating version 38 to 39 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 39 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-09-17T16:02:11.386 INFO:tasks.rados.rados.0.smithi181.stdout:231: left oid 27 (ObjNum 100 snap 24 seq_num 100) 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:228: finishing copy_from to smithi18136759-14 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 60 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:224: finishing copy_from to smithi18136759-13 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 23 (ObjNum 79 snap 17 seq_num 79) dirty exists 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:225: finishing copy_from racing read to smithi18136759-40 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:213: finishing copy_from racing read to smithi18136759-3 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:213: finishing copy_from to smithi18136759-3 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:225: finishing copy_from to smithi18136759-40 2024-09-17T16:02:11.387 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 40 v 24 (ObjNum 85 snap 22 seq_num 85) dirty exists 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:213: done (10 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:221: done (9 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:224: done (8 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:225: done (7 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:227: done (6 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:228: done (5 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:230: done (4 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:231: done (3 left) 2024-09-17T16:02:11.398 INFO:tasks.rados.rados.0.smithi181.stdout:232: done (2 left) 2024-09-17T16:02:11.399 INFO:tasks.rados.rados.0.smithi181.stdout:235: done (1 left) 2024-09-17T16:02:11.399 INFO:tasks.rados.rados.0.smithi181.stdout:236: rmattr oid 43 current snap is 24 2024-09-17T16:02:11.399 INFO:tasks.rados.rados.0.smithi181.stdout:236: done (1 left) 2024-09-17T16:02:11.399 INFO:tasks.rados.rados.0.smithi181.stdout:237: setattr oid 28 current snap is 24 2024-09-17T16:02:11.400 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 17 current snap is 24 2024-09-17T16:02:11.400 INFO:tasks.rados.rados.0.smithi181.stdout:238: seq_num 102 ranges {278528=557056,835584=638976,1474560=524288} 2024-09-17T16:02:11.405 INFO:tasks.rados.rados.0.smithi181.stdout:238: writing smithi18136759-17 from 278528 to 835584 tid 1 2024-09-17T16:02:11.409 INFO:tasks.rados.rados.0.smithi181.stdout:238: writing smithi18136759-17 from 835584 to 1474560 tid 2 2024-09-17T16:02:11.412 INFO:tasks.rados.rados.0.smithi181.stdout:238: writing smithi18136759-17 from 1474560 to 1998848 tid 3 2024-09-17T16:02:11.412 INFO:tasks.rados.rados.0.smithi181.stdout:239: read oid 44 snap -1 2024-09-17T16:02:11.412 INFO:tasks.rados.rados.0.smithi181.stdout:239: expect (ObjNum 88 snap 22 seq_num 88) 2024-09-17T16:02:11.412 INFO:tasks.rados.rados.0.smithi181.stdout:240: snap_remove snap 21 2024-09-17T16:02:11.414 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:11.411+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:11.459 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.459 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:11.515 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:11.513+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:11.615 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.615 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.616 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:11.613+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:11.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:11.716 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:11.714+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:11.730 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.730 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:11.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:11.846 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.846 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.854 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:11.852+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:11.855 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:11.918 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:11.916+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:11.957 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:11.957 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:11.970 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:11.971 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:11.979 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:12.023 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:12.021+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:12.083 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:12.083 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:12.091 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:12.132 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:12.130+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:12.181 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:12.202 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:12.203 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:12.220 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:12.218+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:12.224 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:12.224 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_repop(client.4219.0:41 1.0 e25/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 25'23, mlcod=25'19)", 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 54.309180140000002, 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077219050000000003, 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.236 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923884+0000", 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.237 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923885+0000", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923947+0000", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1790000000000003e-05 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923974+0000", 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7368999999999998e-05 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.238 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.923980+0000", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2469999999999997e-06 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924022+0000", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2015999999999999e-05 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.924037+0000", 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4127e-05 2024-09-17T16:02:12.239 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931579+0000", 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0075422340000000001 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:17.931606+0000", 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7307000000000001e-05 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.240 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 30.325841616000002, 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.241 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:02:12.242 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.243 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:02:12.244 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 30.324371705000001, 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.245 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.246 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:02:12.247 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.248 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.318018989999999, 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:12.249 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:12.250 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.251 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:02:12.252 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.253 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.254 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:02:12.255 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.316036402999998, 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:12.256 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.257 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:02:12.259 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.260 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:02:12.261 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:02:12.262 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.263 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.315610535000001, 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:12.264 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.265 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:02:12.266 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.267 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.268 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:02:12.269 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.270 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.315582365000001, 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:12.271 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.272 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.273 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:02:12.275 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.276 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.4s1 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.304419209999999, 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077955969999999996, 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.277 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928645+0000", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928646+0000", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928660+0000", 2024-09-17T16:02:12.278 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4535000000000001e-05 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928684+0000", 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3938e-05 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928692+0000", 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5009999999999998e-06 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.279 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928739+0000", 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7574999999999998e-05 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.928746+0000", 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9929999999999998e-06 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936413+0000", 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076669930000000004 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.280 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.936441+0000", 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7413999999999999e-05 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.302781511999999, 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.281 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.282 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:02:12.283 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.284 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.285 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.302461885, 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.286 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:02:12.287 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.288 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.289 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.290 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.302370070999999, 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.291 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:02:12.292 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:02:12.293 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.294 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=30, last_complete=46'18, committed=1, applied=1))", 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.312976768999999, 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24942447400000001, 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.295 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:12.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920091+0000", 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6620000000000001e-06 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168910+0000", 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24881899499999999 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168924+0000", 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3968000000000001e-05 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168997+0000", 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3919999999999997e-05 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.169512+0000", 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051455700000000001 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.311764921999998, 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080140109999999997, 2024-09-17T16:02:12.299 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921300+0000", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921319+0000", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8519e-05 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2544999999999998e-05 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921346+0000", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9420000000000003e-06 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921389+0000", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2620000000000002e-05 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921394+0000", 2024-09-17T16:02:12.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3519999999999997e-06 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929283+0000", 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007889007 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929313+0000", 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0460999999999999e-05 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.308059829000001, 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010009911, 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:12.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925020+0000", 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925042+0000", 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2407000000000001e-05 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925050+0000", 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3089999999999998e-06 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925097+0000", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7763999999999999e-05 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925103+0000", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0759999999999999e-06 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098665709999999993 2024-09-17T16:02:12.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935014+0000", 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4459000000000002e-05 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.308 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.303289709000001, 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078830700000000007, 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:12.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929783+0000", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3150000000000003e-06 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929802+0000", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9959999999999999e-05 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929810+0000", 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3069999999999997e-06 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929880+0000", 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0123000000000006e-05 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929890+0000", 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937625+0000", 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077345599999999997 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937658+0000", 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2662e-05 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s5 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:12.313 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.8940661490000004, 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092966230000000004, 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338999+0000", 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1199999999999999e-07 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339027+0000", 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8099999999999999e-05 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339113+0000", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5958000000000006e-05 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339121+0000", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5949999999999999e-06 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339202+0000", 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1150999999999996e-05 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339213+0000", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0607e-05 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348258+0000", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090445890000000004 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348295+0000", 2024-09-17T16:02:12.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7311000000000002e-05 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.8909257630000003, 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092835030000000002, 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342160+0000", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0803999999999999e-05 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342475+0000", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031531799999999998 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342484+0000", 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1609999999999992e-06 2024-09-17T16:02:12.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342552+0000", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7761000000000004e-05 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342562+0000", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0316000000000001e-05 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351383+0000", 2024-09-17T16:02:12.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088202019999999992 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351422+0000", 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9530000000000003e-05 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.13s4 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.886795867, 2024-09-17T16:02:12.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081316900000000004, 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346269+0000", 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346318+0000", 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8843e-05 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346402+0000", 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4646000000000002e-05 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346411+0000", 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3890000000000005e-06 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346518+0000", 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107693 2024-09-17T16:02:12.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346525+0000", 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5200000000000003e-06 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354365+0000", 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078396709999999994 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354400+0000", 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5627999999999998e-05 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 64/32 ECSubWrite(tid=68, reqid=osd.0.0:68, at_version=64'32, trim_to=0'0, roll_forward_to=62'30))", 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.8866950960000004, 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081747250000000007, 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.327 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346370+0000", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6e-07 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346392+0000", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2419e-05 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346501+0000", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010902 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346509+0000", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0609999999999996e-06 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346569+0000", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0405999999999997e-05 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346579+0000", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4700000000000008e-06 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354505+0000", 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079256489999999999 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354544+0000", 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9440000000000002e-05 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 68/32 ECSubWriteReply(tid=68, last_complete=68'49, committed=1, applied=1))", 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:12.331 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.8393346160000004, 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44479439100000001, 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7499999999999999e-07 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393734+0000", 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9659999999999998e-06 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837925+0000", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44419083199999998 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837936+0000", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0349e-05 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838002+0000", 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6196000000000001e-05 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838524+0000", 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052227299999999996 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.335 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:12.336 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:12.337 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:12.337 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:12.338 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:12.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:12.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:12.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:12.339 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:12.340 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:12.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:12.340 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:12.334+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:12.349 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:12.350 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:12.371 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:12.371 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:12.385 INFO:tasks.rados.rados.0.smithi181.stdout:240: done (4 left) 2024-09-17T16:02:12.385 INFO:tasks.rados.rados.0.smithi181.stdout:241: read oid 25 snap -1 2024-09-17T16:02:12.385 INFO:tasks.rados.rados.0.smithi181.stdout:241: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T16:02:12.385 INFO:tasks.rados.rados.0.smithi181.stdout:242: snap_create 2024-09-17T16:02:12.385 INFO:tasks.rados.rados.0.smithi181.stdout:234: expect (ObjNum 94 snap 23 seq_num 94) 2024-09-17T16:02:12.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:12.389 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 689378 2024-09-17T16:02:12.394 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:12.395 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:12.400 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.416 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.418 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:12.418 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:12.430 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 39 (ObjNum 4113 snap 0 seq_num 2549451346) dirty exists 2024-09-17T16:02:12.430 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing write tid 1 to smithi18136759-17 2024-09-17T16:02:12.430 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing write tid 2 to smithi18136759-17 2024-09-17T16:02:12.430 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing write tid 3 to smithi18136759-17 2024-09-17T16:02:12.431 INFO:tasks.rados.rados.0.smithi181.stdout:239: expect (ObjNum 88 snap 22 seq_num 88) 2024-09-17T16:02:12.431 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.437 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:12.434+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:12.438 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:12.439 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:12.439 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:12.437+0000 7f4536ffd640 0 -- 172.21.15.6:0/2112710288 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f45382218a0 msgr2=0x7f4538223c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:12.452 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:12.452 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:12.452 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:12.452 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:12.461 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:12.472 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.514 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.538 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:12.536+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing write tid 4 to smithi18136759-17 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: finishing write tid 5 to smithi18136759-17 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: oid 17 updating version 0 to 38 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: oid 17 version 38 is already newer than 35 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: oid 17 version 38 is already newer than 37 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: oid 17 version 38 is already newer than 36 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 17 v 38 (ObjNum 102 snap 24 seq_num 102) dirty exists 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:238: left oid 17 (ObjNum 102 snap 24 seq_num 102) 2024-09-17T16:02:12.556 INFO:tasks.rados.rados.0.smithi181.stdout:241: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T16:02:12.558 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 385892 2024-09-17T16:02:12.558 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.562 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:12.564 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:12.564 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:12.564 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:12.564 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:12.564 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:12.572 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:12.639 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:12.637+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:12.662 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:12.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:12.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:12.719 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 35.831611770000002, 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038886110000000002, 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.720 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884707+0000", 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2036999999999999e-05 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.721 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884745+0000", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7843999999999999e-05 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884756+0000", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1240000000000001e-05 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884826+0000", 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9873e-05 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.722 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884836+0000", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0053e-05 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888486+0000", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036497270000000002 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888573+0000", 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7068000000000001e-05 2024-09-17T16:02:12.723 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 35.830001975000002, 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.724 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.725 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:02:12.726 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:02:12.727 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.728 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 35.829190732000001, 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044954000000000001, 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:12.729 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.730 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4892000000000002e-05 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887167+0000", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5617999999999997e-05 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887176+0000", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6379999999999993e-06 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.731 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887237+0000", 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1760000000000005e-05 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887246+0000", 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891572+0000", 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004325626 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.732 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891601+0000", 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9105e-05 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.792197098999999, 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054480279999999997, 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.733 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924122+0000", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2844999999999999e-05 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924159+0000", 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6534e-05 2024-09-17T16:02:12.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924166+0000", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.742e-06 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924210+0000", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4641999999999997e-05 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924220+0000", 2024-09-17T16:02:12.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7580000000000004e-06 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929512+0000", 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005292265 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929547+0000", 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5046999999999998e-05 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.737 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=51, reqid=osd.12.0:51, at_version=46'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.792089960999999, 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053997719999999997, 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924207+0000", 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924223+0000", 2024-09-17T16:02:12.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6835000000000001e-05 2024-09-17T16:02:12.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924252+0000", 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8285000000000001e-05 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924259+0000", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924320+0000", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1709000000000002e-05 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924329+0000", 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8799999999999997e-06 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929582+0000", 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052531169999999999 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929606+0000", 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3913e-05 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.791267721000001, 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:02:12.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 46/32 ECSubWrite(tid=70, reqid=osd.13.0:67, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.790940334999998, 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042172039999999996, 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.748 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925357+0000", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925366+0000", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925396+0000", 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9964e-05 2024-09-17T16:02:12.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925404+0000", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6429999999999995e-06 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925470+0000", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6817999999999996e-05 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925480+0000", 2024-09-17T16:02:12.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929543+0000", 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040623580000000003 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929573+0000", 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0682000000000002e-05 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.788908759000002, 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:12.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:02:12.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.755 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.786508117, 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052541769999999996, 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.757 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929789+0000", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929800+0000", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1341e-05 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929829+0000", 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8642000000000001e-05 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929837+0000", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929901+0000", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4067999999999997e-05 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929907+0000", 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.609e-06 2024-09-17T16:02:12.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934990+0000", 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050827579999999997 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935042+0000", 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2676999999999997e-05 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=39, reqid=osd.0.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.783747963, 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050006720000000003, 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932549+0000", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932561+0000", 2024-09-17T16:02:12.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1674e-05 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932581+0000", 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0024999999999999e-05 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932590+0000", 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7780000000000002e-06 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932642+0000", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1753999999999999e-05 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932651+0000", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2310000000000002e-06 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937518+0000", 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048669969999999996 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937549+0000", 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0651999999999997e-05 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.365039881000001, 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043350350000000001, 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.766 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.767 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351257+0000", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0100000000000001e-07 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351295+0000", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8333e-05 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351337+0000", 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1491e-05 2024-09-17T16:02:12.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351345+0000", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5189999999999992e-06 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351424+0000", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351434+0000", 2024-09-17T16:02:12.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355546+0000", 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041123100000000001 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355591+0000", 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5212e-05 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.770 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=85, reqid=osd.13.0:78, at_version=0'0, trim_to=0'0, roll_forward_to=52'18))", 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.34595508, 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037161669999999998, 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:12.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.17e-07 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370349+0000", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4499999999999998e-06 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370413+0000", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4089000000000003e-05 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370419+0000", 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2740000000000004e-06 2024-09-17T16:02:12.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370550+0000", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131163 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370561+0000", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374029+0000", 2024-09-17T16:02:12.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034684590000000001 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374057+0000", 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8064000000000001e-05 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.12s3 64/61 ECSubWrite(tid=42, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=64'34))", 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345060+0000", 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.371236141000001, 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036823530000000002, 2024-09-17T16:02:12.775 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345060+0000", 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345060+0000", 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345060+0000", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345068+0000", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9189999999999999e-06 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345163+0000", 2024-09-17T16:02:12.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4128000000000004e-05 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345172+0000", 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9199999999999999e-06 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345290+0000", 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011756099999999999 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345297+0000", 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.109e-06 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348698+0000", 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034009090000000001 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348743+0000", 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4530000000000002e-05 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.779 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.370559037, 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060163380000000004, 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8900000000000001e-07 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345763+0000", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5772e-05 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345830+0000", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6855999999999998e-05 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345839+0000", 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:02:12.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345913+0000", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3454999999999994e-05 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345924+0000", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351719+0000", 2024-09-17T16:02:12.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057956309999999999 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351754+0000", 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4439e-05 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:12.784 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.370530207, 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087141639999999999, 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345787+0000", 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0109e-05 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346312+0000", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052588999999999999 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346324+0000", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1445e-05 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346394+0000", 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9604000000000006e-05 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346403+0000", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.611e-06 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354448+0000", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080448199999999994 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354480+0000", 2024-09-17T16:02:12.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2329000000000002e-05 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.370097869, 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082305039999999996, 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346199+0000", 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7300000000000002e-07 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346223+0000", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4442000000000001e-05 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346360+0000", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136792 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346374+0000", 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3669000000000001e-05 2024-09-17T16:02:12.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346429+0000", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4898999999999997e-05 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346437+0000", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354393+0000", 2024-09-17T16:02:12.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079559079999999994 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354429+0000", 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6168999999999999e-05 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.10s1 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:12.793 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.370058867999999, 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083156649999999999, 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346238+0000", 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346256+0000", 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8805999999999999e-05 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346408+0000", 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000151519 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346417+0000", 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6619999999999994e-06 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346513+0000", 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6127999999999998e-05 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346523+0000", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9750000000000002e-06 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354521+0000", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079977780000000005 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354553+0000", 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2614000000000003e-05 2024-09-17T16:02:12.797 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 64/32 ECSubWrite(tid=130, reqid=osd.12.0:127, at_version=0'0, trim_to=0'0, roll_forward_to=64'43))", 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.369079594, 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045663939999999997, 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5499999999999998e-07 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347225+0000", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7279999999999995e-06 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347280+0000", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5435e-05 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347288+0000", 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4510000000000003e-06 2024-09-17T16:02:12.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347350+0000", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2533000000000006e-05 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347359+0000", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4349999999999997e-06 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351754+0000", 2024-09-17T16:02:12.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043954759999999997 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351783+0000", 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8881e-05 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 69/53 ECSubWrite(tid=216, reqid=client.4560.0:480, at_version=69'34, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.3695619069999996, 2024-09-17T16:02:12.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049892540000000003, 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346735+0000", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5099999999999999e-07 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347399+0000", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066419799999999996 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347439+0000", 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9925000000000001e-05 2024-09-17T16:02:12.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347447+0000", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7159999999999992e-06 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347504+0000", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6396e-05 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347514+0000", 2024-09-17T16:02:12.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0526999999999999e-05 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351687+0000", 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041724650000000002 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351724+0000", 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6875999999999997e-05 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.806 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 69/53 ECSubWrite(tid=217, reqid=client.4560.0:481, at_version=69'35, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.3656273130000001, 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039025790000000002, 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:12.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350670+0000", 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350992+0000", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321758 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351026+0000", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4221000000000003e-05 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351036+0000", 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0044000000000001e-05 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351104+0000", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8035999999999997e-05 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351114+0000", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9059999999999997e-06 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.354535+0000", 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034212610000000001 2024-09-17T16:02:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.354572+0000", 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6452000000000003e-05 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:12.811 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:12.812 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:12.812 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:12.812 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:02:12.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:02:12.813 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:12.810+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:12.814 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:12.913 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:12.910+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:12.918 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:12.918 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:12.918 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:12.918 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:12.927 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:12.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-17T16:02:13.013 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:13.011+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:13.030 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:13.030 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:13.030 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:13.030 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:13.031 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:13.039 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:13.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:02:13.114 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:13.112+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.144 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.286544638999999, 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.145 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.146 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.147 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.148 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.285643765000003, 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.149 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:02:13.150 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:02:13.151 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.152 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:13.153 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.281553938000002, 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.154 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.155 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.156 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.157 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=5, reqid=client.4560.0:38, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.280215491, 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055595449999999999, 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.158 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861312+0000", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861313+0000", 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9800000000000002e-07 2024-09-17T16:02:13.159 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861333+0000", 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0276999999999999e-05 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861372+0000", 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9365e-05 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.160 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861382+0000", 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3349999999999996e-06 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861456+0000", 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3972000000000003e-05 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.861465+0000", 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7480000000000005e-06 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.161 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866841+0000", 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053751720000000001 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866871+0000", 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0678e-05 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.162 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=1, reqid=client.4560.0:31, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.279250390999998, 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045470969999999999, 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.163 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862277+0000", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862278+0000", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5199999999999995e-07 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862373+0000", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5013999999999995e-05 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862431+0000", 2024-09-17T16:02:13.164 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8057000000000003e-05 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862440+0000", 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6339999999999996e-06 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862513+0000", 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3028999999999995e-05 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.165 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862522+0000", 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8859999999999993e-06 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866779+0000", 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042571140000000002 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866824+0000", 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4511000000000001e-05 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.166 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.279146977000003, 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.167 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.168 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.169 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:02:13.170 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.277321598999997, 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.171 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.172 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.173 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.174 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.277173523999998, 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.175 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.176 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:02:13.177 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:02:13.178 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.276523640000001, 2024-09-17T16:02:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.180 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:02:13.181 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:02:13.182 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.183 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.273946277, 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.184 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.185 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:02:13.186 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:02:13.187 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.273455104999996, 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:13.188 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.189 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.190 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:02:13.191 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 35/32 ECSubWrite(tid=6, reqid=client.4560.0:43, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.272718253999997, 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048136530000000002, 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.192 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868809+0000", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.193 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868810+0000", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9099999999999991e-07 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868951+0000", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014129799999999999 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868987+0000", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5556000000000003e-05 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.194 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868996+0000", 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6549999999999999e-06 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869062+0000", 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5714000000000001e-05 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.869072+0000", 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0273e-05 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.195 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873569+0000", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044970189999999997 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873623+0000", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3146999999999998e-05 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 46/32 ECSubWrite(tid=40, reqid=osd.0.0:37, at_version=0'0, trim_to=0'0, roll_forward_to=46'12))", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:02:13.196 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.206051684999998, 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005398712, 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935475+0000", 2024-09-17T16:02:13.197 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935476+0000", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6000000000000004e-07 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935486+0000", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6299999999999993e-06 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935508+0000", 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2333000000000001e-05 2024-09-17T16:02:13.198 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935516+0000", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8779999999999998e-06 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935579+0000", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3529999999999997e-05 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935588+0000", 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9560000000000003e-06 2024-09-17T16:02:13.199 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940834+0000", 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052456009999999999 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.940874+0000", 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0224e-05 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.200 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 52/32 ECSubWrite(tid=82, reqid=client.4560.0:335, at_version=52'11, trim_to=0'0, roll_forward_to=52'8))", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.777778493, 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048268470000000004, 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.201 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363749+0000", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8800000000000002e-07 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363764+0000", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4596000000000001e-05 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.202 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363789+0000", 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4715e-05 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363795+0000", 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0619999999999997e-06 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363861+0000", 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6214e-05 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.203 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.363873+0000", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2153e-05 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.368548+0000", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046749959999999998 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.368576+0000", 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7423000000000002e-05 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.204 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 52/32 ECSubWrite(tid=84, reqid=client.4560.0:335, at_version=0'0, trim_to=0'0, roll_forward_to=52'11))", 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.77238964, 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048528909999999998, 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.205 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369138+0000", 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1200000000000003e-07 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.206 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.369146+0000", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7160000000000003e-06 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371657+0000", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025113610000000001 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371664+0000", 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6869999999999997e-06 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.207 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371713+0000", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8724999999999999e-05 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.371719+0000", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-06 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373960+0000", 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022413239999999998 2024-09-17T16:02:13.208 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373990+0000", 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0496000000000001e-05 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 52/32 ECSubWrite(tid=89, reqid=client.4560.0:340, at_version=52'11, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 22.768533694999999, 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15958299200000001, 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.209 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.210 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372994+0000", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373011+0000", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6280999999999999e-05 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531935+0000", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15892432300000001 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531949+0000", 2024-09-17T16:02:13.211 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532026+0000", 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6526000000000006e-05 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532033+0000", 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.212 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532538+0000", 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050494099999999996 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532576+0000", 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8059999999999998e-05 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.213 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 54/32 ECSubWrite(tid=53, reqid=client.4560.0:344, at_version=54'24, trim_to=0'0, roll_forward_to=52'22))", 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.798860570999999, 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045033649999999996, 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.214 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342667+0000", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342688+0000", 2024-09-17T16:02:13.215 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1129e-05 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342910+0000", 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00022252599999999999 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342920+0000", 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4749999999999999e-06 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.216 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342980+0000", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0260999999999998e-05 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.342990+0000", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4709999999999996e-06 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.347131+0000", 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041411149999999999 2024-09-17T16:02:13.217 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.347170+0000", 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9205000000000002e-05 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 54/32 ECSubWrite(tid=54, reqid=client.4560.0:344, at_version=0'0, trim_to=0'0, roll_forward_to=54'24))", 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.793321958, 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0046180140000000001, 2024-09-17T16:02:13.218 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348205+0000", 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.219 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348206+0000", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7700000000000004e-07 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348215+0000", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.268e-06 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348242+0000", 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7356e-05 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.220 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348252+0000", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0018e-05 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348318+0000", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5888000000000003e-05 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.348330+0000", 2024-09-17T16:02:13.221 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1232e-05 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.352784+0000", 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044541750000000003 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:52.352823+0000", 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9499999999999998e-05 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.222 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 10.803009225, 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058173629999999999, 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.223 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5199999999999997e-07 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338580+0000", 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1075999999999996e-05 2024-09-17T16:02:13.224 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338633+0000", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3263999999999997e-05 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338646+0000", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3326000000000001e-05 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339218+0000", 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057138000000000002 2024-09-17T16:02:13.225 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339229+0000", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1487e-05 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344292+0000", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050634460000000001 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344335+0000", 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2831999999999999e-05 2024-09-17T16:02:13.226 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 69/53 ECSubWrite(tid=97, reqid=client.4560.0:497, at_version=69'23, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.376670+0000", 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.7648576509999998, 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053129320000000002, 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.227 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376670+0000", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376670+0000", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376670+0000", 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0800000000000004e-07 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.228 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376684+0000", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4126e-05 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376716+0000", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1446999999999999e-05 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376723+0000", 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5050000000000001e-06 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.229 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376776+0000", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2706e-05 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376785+0000", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3940000000000002e-06 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.381959+0000", 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051737170000000004 2024-09-17T16:02:13.230 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.381982+0000", 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3428999999999999e-05 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.231 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:13.233 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:13.233 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:13.230+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:13.333 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:13.331+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:13.377 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:13.377 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:13.377 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:13.377 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:13.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-17T16:02:13.386 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:13.389 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:234: done (5 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:237: done (4 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:238: done (3 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:239: done (2 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:241: done (1 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:242: done (0 left) 2024-09-17T16:02:13.395 INFO:tasks.rados.rados.0.smithi181.stdout:243: setattr oid 18 current snap is 25 2024-09-17T16:02:13.397 INFO:tasks.rados.rados.0.smithi181.stdout:244: delete oid 22 current snap is 25 2024-09-17T16:02:13.404 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 130 (ObjNum 94 snap 23 seq_num 94) dirty exists 2024-09-17T16:02:13.406 INFO:tasks.rados.rados.0.smithi181.stdout:243: done (1 left) 2024-09-17T16:02:13.406 INFO:tasks.rados.rados.0.smithi181.stdout:244: done (0 left) 2024-09-17T16:02:13.406 INFO:tasks.rados.rados.0.smithi181.stdout:245: read oid 40 snap -1 2024-09-17T16:02:13.406 INFO:tasks.rados.rados.0.smithi181.stdout:245: expect (ObjNum 85 snap 22 seq_num 85) 2024-09-17T16:02:13.406 INFO:tasks.rados.rados.0.smithi181.stdout:246: snap_create 2024-09-17T16:02:13.427 INFO:tasks.rados.rados.0.smithi181.stdout:245: expect (ObjNum 85 snap 22 seq_num 85) 2024-09-17T16:02:13.427 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:13.447 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:13.445+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:13.491 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:13.492 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:13.492 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:13.492 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:13.492 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:13.496 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:13.500 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:13.544 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:13.542+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:13.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:02:13.574 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1850282 2024-09-17T16:02:13.604 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:13.604 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:13.604 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.749167454000002, 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:02:13.605 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.606 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:02:13.607 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:02:13.608 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.609 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.749138940999998, 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:13.610 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.611 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.612 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:02:13.613 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:02:13.614 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.615 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.748999896000001, 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.616 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.617 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:02:13.618 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.619 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.620 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.748972221999999, 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:02:13.621 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.622 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:02:13.623 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.624 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:02:13.625 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.626 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.684535644, 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.627 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.628 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:02:13.629 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:02:13.630 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.631 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.632 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.684510439999997, 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.633 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:02:13.634 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:02:13.635 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.636 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:02:13.637 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.681144445999998, 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.638 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:02:13.639 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.640 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.641 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.654169797000002, 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.642 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.643 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.644 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:02:13.645 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.646 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:02:13.647 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.654151925999997, 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.648 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.649 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:02:13.650 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.651 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.652 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.653 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 39.650858159000002, 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.654 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.655 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:02:13.656 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.657 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 30.650742403999999, 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.658 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.659 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:02:13.660 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.661 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.670889024000001, 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.662 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:02:13.663 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.664 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.665 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 55/53 ECSubWriteReply(tid=63, last_complete=55'29, committed=1, applied=1))", 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.224973785, 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.850069713, 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.666 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376978+0000", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376981+0000", 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:02:13.667 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226831+0000", 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84985070900000004 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226846+0000", 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4192999999999999e-05 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.668 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226976+0000", 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013074699999999999 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.227047+0000", 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0347999999999998e-05 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:13.669 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 20.208774289000001, 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023393622999999999, 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.670 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.82e-07 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393970+0000", 2024-09-17T16:02:13.671 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079326400000000001 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414706+0000", 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020736065000000001 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414712+0000", 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.672 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414788+0000", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6206999999999997e-05 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414798+0000", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416538+0000", 2024-09-17T16:02:13.673 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00174007 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416570+0000", 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2415000000000001e-05 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpReadReply(3.0s0 68/53 ECSubReadReply(tid=92, attrs_read=0))", 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.7436270140000003, 2024-09-17T16:02:13.674 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.495310574, 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:13.675 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.859357+0000", 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010325270000000001 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.676 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352866+0000", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49350909799999998 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352872+0000", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5819999999999998e-06 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352920+0000", 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7234999999999998e-05 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.677 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.353634+0000", 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071472599999999995 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 69/53 ECSubWriteReply(tid=93, last_complete=69'46, committed=1, applied=1))", 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.2491122849999998, 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02328291, 2024-09-17T16:02:13.678 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.679 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4e-07 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352844+0000", 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6619999999999996e-06 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.680 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375190+0000", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022345941000000001 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375196+0000", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375242+0000", 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6655e-05 2024-09-17T16:02:13.681 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376121+0000", 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087903899999999997 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:497 3.9s0 3:905994b3:::smithi18136759-13:head [append 0~0] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:13.682 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.228430608, 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.035534871000000003, 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 497 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.683 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373521+0000", 2024-09-17T16:02:13.684 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8200000000000001e-07 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373523+0000", 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.272e-06 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373529+0000", 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2750000000000001e-06 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.685 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373534+0000", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0359999999999997e-06 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375754+0000", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002219834 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375866+0000", 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011218900000000001 2024-09-17T16:02:13.686 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376638+0000", 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077170400000000001 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380515+0000", 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038770889999999998 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:13.687 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408971+0000", 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02845599 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409055+0000", 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4099999999999998e-05 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 69/53 ECSubWriteReply(tid=97, last_complete=69'23, committed=1, applied=1))", 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:13.688 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.2216866680000003, 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029135165000000001, 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.689 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380265+0000", 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.690 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408819+0000", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028553388999999998 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408826+0000", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0140000000000001e-06 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408882+0000", 2024-09-17T16:02:13.691 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5704000000000001e-05 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409399+0000", 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051776899999999995 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 69/53 ECSubWriteReply(tid=104, last_complete=69'50, committed=1, applied=1))", 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:13.692 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.1822162020000002, 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35452754399999997, 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.693 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419738+0000", 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.65e-06 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.694 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774110+0000", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35437182699999997 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774122+0000", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2642e-05 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774218+0000", 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5758e-05 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.695 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774262+0000", 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4205000000000001e-05 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 74/53 ECSubWriteReply(tid=138, last_complete=74'39, committed=1, applied=1))", 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.1973540709999999, 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36342225099999997, 2024-09-17T16:02:13.696 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:13.697 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8000000000000002e-07 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404599+0000", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7549999999999999e-06 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767461+0000", 2024-09-17T16:02:13.698 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36286232000000002 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767477+0000", 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5934e-05 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767592+0000", 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011500199999999999 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:13.699 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.768019+0000", 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042695999999999999 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:13.700 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:13.701 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:13.699+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:13.702 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:13.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-17T16:02:13.804 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:13.804 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:13.804 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:13.804 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:13.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:13.817 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:13.815+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:13.904 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:13.902+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:13.915 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:13.916 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:13.916 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:13.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:02:13.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:14.006 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:14.004+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=167, reqid=client.4560.0:228, at_version=44'14, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:14.028 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.117137434, 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045964070000000003, 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908482+0000", 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908499+0000", 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6748999999999998e-05 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908533+0000", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3831999999999997e-05 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908539+0000", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5980000000000002e-06 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908619+0000", 2024-09-17T16:02:14.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740999999999994e-05 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908629+0000", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913043+0000", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044149380000000002 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913078+0000", 2024-09-17T16:02:14.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4094999999999998e-05 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 31.097421260000001, 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003430879, 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.033 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928197+0000", 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928198+0000", 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2200000000000002e-07 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928663+0000", 2024-09-17T16:02:14.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046509200000000001 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928719+0000", 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5677000000000002e-05 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928728+0000", 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6729999999999998e-06 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928810+0000", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2386999999999996e-05 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.928821+0000", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0797e-05 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931371+0000", 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002550549 2024-09-17T16:02:14.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.931628+0000", 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000256782 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.097874253000001, 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046997530000000001, 2024-09-17T16:02:14.038 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927762+0000", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7229000000000001e-05 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927945+0000", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018362200000000001 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928161+0000", 2024-09-17T16:02:14.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021584100000000001 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928170+0000", 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6249999999999996e-06 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928242+0000", 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2434999999999993e-05 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928253+0000", 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932409+0000", 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041564169999999999 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932444+0000", 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.042 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.097669227000001, 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048629329999999998, 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:14.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927950+0000", 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3399999999999999e-07 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928041+0000", 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0953999999999997e-05 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928205+0000", 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000164537 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928213+0000", 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.272e-06 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929483+0000", 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001270106 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929493+0000", 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0207000000000001e-05 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932778+0000", 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032847419999999998 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932812+0000", 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4581e-05 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.047 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=12, reqid=client.4560.0:273, at_version=45'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.097537708000001, 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043524990000000001, 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928082+0000", 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2799999999999996e-07 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928098+0000", 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6070999999999999e-05 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928229+0000", 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013130999999999999 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928234+0000", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928583+0000", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034869300000000002 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928595+0000", 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1433e-05 2024-09-17T16:02:14.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932398+0000", 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038034890000000002 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932433+0000", 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5386999999999998e-05 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.052 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=13, reqid=client.4560.0:274, at_version=45'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.096877321000001, 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.003718285, 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928741+0000", 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928742+0000", 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4799999999999998e-07 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928761+0000", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8666000000000001e-05 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928789+0000", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8535999999999999e-05 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928798+0000", 2024-09-17T16:02:14.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4479999999999994e-06 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928947+0000", 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000149068 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928956+0000", 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0799999999999995e-06 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932437+0000", 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034817609999999999 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932460+0000", 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2078000000000001e-05 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.057 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.107075971, 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0039399220000000002, 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918543+0000", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.29e-07 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5406999999999999e-05 2024-09-17T16:02:14.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000102031 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918679+0000", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918809+0000", 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012967600000000001 2024-09-17T16:02:14.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918820+0000", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1156e-05 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922452+0000", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036319080000000001 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922483+0000", 2024-09-17T16:02:14.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0338999999999999e-05 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.106649872999999, 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035116600000000002, 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918969+0000", 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.918994+0000", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4694000000000001e-05 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919035+0000", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1223000000000002e-05 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919044+0000", 2024-09-17T16:02:14.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0599999999999997e-06 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919108+0000", 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3326999999999999e-05 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919118+0000", 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0037999999999999e-05 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922439+0000", 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033215340000000001 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.922480+0000", 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1010000000000002e-05 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:14.066 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.106021024, 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043836329999999996, 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919621+0000", 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2204e-05 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919652+0000", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1846e-05 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919661+0000", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:02:14.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3075000000000002e-05 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919733+0000", 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3540000000000006e-06 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923937+0000", 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042032980000000003 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923981+0000", 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4685000000000003e-05 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=79, reqid=client.4560.0:332, at_version=52'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.668004195999998, 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051054819999999997, 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.071 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357615+0000", 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-07 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358043+0000", 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042830599999999999 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358075+0000", 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2512999999999999e-05 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358085+0000", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0081000000000001e-05 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358170+0000", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4362000000000003e-05 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358181+0000", 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1535000000000001e-05 2024-09-17T16:02:14.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362680+0000", 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989069999999999 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362720+0000", 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9641000000000002e-05 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=81, reqid=client.4560.0:334, at_version=52'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 23.662838356999998, 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033175610000000001, 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:14.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362780+0000", 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362781+0000", 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.71e-07 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362824+0000", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3049000000000001e-05 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362863+0000", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9063999999999997e-05 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362873+0000", 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4199999999999996e-06 2024-09-17T16:02:14.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.363453+0000", 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00057987999999999996 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.363463+0000", 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0287e-05 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.366063+0000", 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026005630000000002 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.366098+0000", 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4427000000000001e-05 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=197, reqid=client.4560.0:392, at_version=60'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.080 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.675246688, 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0099856809999999997, 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2599999999999999e-07 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350537+0000", 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016478400000000001 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350604+0000", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7305999999999996e-05 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350615+0000", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1154e-05 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350701+0000", 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6136000000000003e-05 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350721+0000", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9967e-05 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360314+0000", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0095931230000000003 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360358+0000", 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3084999999999998e-05 2024-09-17T16:02:14.084 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=198, reqid=client.4560.0:393, at_version=60'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.672720972, 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075206600000000002, 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352899+0000", 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1230000000000001e-06 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353041+0000", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014228299999999999 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353091+0000", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9942999999999999e-05 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353101+0000", 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:02:14.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353167+0000", 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5365999999999997e-05 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353178+0000", 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1429e-05 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360364+0000", 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071857500000000003 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360418+0000", 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4317000000000002e-05 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=199, reqid=client.4560.0:394, at_version=60'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.089 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.671032421, 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058628329999999996, 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354587+0000", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4099999999999997e-07 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354652+0000", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5085000000000005e-05 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354702+0000", 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9641999999999997e-05 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354711+0000", 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0869999999999996e-06 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354774+0000", 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3048999999999999e-05 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354784+0000", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163000000000001e-05 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360427+0000", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056429729999999999 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360449+0000", 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1993000000000001e-05 2024-09-17T16:02:14.093 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=200, reqid=client.4560.0:395, at_version=60'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 15.669606282, 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989890000000001, 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356013+0000", 2024-09-17T16:02:14.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356031+0000", 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7592999999999999e-05 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356068+0000", 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7345999999999998e-05 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356079+0000", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0730999999999999e-05 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356150+0000", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1668999999999994e-05 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356161+0000", 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1027e-05 2024-09-17T16:02:14.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360478+0000", 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043165800000000004 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360511+0000", 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3251999999999999e-05 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.687362568999999, 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046971620000000004, 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338276+0000", 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9932000000000002e-05 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338298+0000", 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1336e-05 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338306+0000", 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6810000000000004e-06 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338467+0000", 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160538 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338477+0000", 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5929999999999995e-06 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.102 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342913+0000", 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044366149999999997 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342953+0000", 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0061000000000002e-05 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.103 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=134, reqid=osd.13.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=64'15))", 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.675571809000001, 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042510439999999998, 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350056+0000", 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:02:14.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350080+0000", 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4360999999999999e-05 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350089+0000", 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2609999999999993e-06 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350189+0000", 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100868 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350199+0000", 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1679999999999993e-06 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354267+0000", 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040685629999999999 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354298+0000", 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0626000000000001e-05 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 64/53 ECSubWrite(tid=211, reqid=osd.11.0:211, at_version=64'37, trim_to=0'0, roll_forward_to=62'35))", 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 11.666552437, 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048593489999999998, 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.108 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359067+0000", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6699999999999999e-07 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359087+0000", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0594999999999998e-05 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359129+0000", 2024-09-17T16:02:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1458999999999997e-05 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359138+0000", 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6050000000000004e-06 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359207+0000", 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8885000000000002e-05 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359218+0000", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0524e-05 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363901+0000", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046832100000000002 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363925+0000", 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4703999999999999e-05 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.112 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.6745937900000003, 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059954810000000004, 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:14.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351050+0000", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4579e-05 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351200+0000", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015020800000000001 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351208+0000", 2024-09-17T16:02:14.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6799999999999999e-06 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352434+0000", 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012252350000000001 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352445+0000", 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1440000000000001e-05 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.356973+0000", 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045277900000000003 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.357020+0000", 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=102, reqid=client.4560.0:502, at_version=69'50, trim_to=0'0, roll_forward_to=69'46))", 2024-09-17T16:02:14.117 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 6.6156128020000002, 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042988599999999998, 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.37e-07 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410021+0000", 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4741e-05 2024-09-17T16:02:14.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410042+0000", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0798e-05 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410047+0000", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5459999999999999e-06 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410084+0000", 2024-09-17T16:02:14.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6556e-05 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410089+0000", 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4600000000000002e-06 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414267+0000", 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041775720000000001 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414305+0000", 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.765e-05 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.122 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:02:14.124 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:14.121+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:14.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:14.224 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:14.222+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.228 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:14.228 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:14.228 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:14.278 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:14.281 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:14.281 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-17T16:02:14.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:02:14.325 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:14.323+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:14.338 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.339 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:14.339 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:14.339 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:14.339 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:14.396 INFO:tasks.rados.rados.0.smithi181.stdout:245: done (1 left) 2024-09-17T16:02:14.396 INFO:tasks.rados.rados.0.smithi181.stdout:246: done (0 left) 2024-09-17T16:02:14.396 INFO:tasks.rados.rados.0.smithi181.stdout:247: rmattr oid 13 current snap is 26 2024-09-17T16:02:14.396 INFO:tasks.rados.rados.0.smithi181.stdout:247: done (0 left) 2024-09-17T16:02:14.396 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 18 current snap is 26 2024-09-17T16:02:14.397 INFO:tasks.rados.rados.0.smithi181.stdout:248: seq_num 104 ranges {1589248=589824,2179072=540672,2719744=622592} 2024-09-17T16:02:14.408 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18136759-18 from 1589248 to 2179072 tid 1 2024-09-17T16:02:14.411 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18136759-18 from 2179072 to 2719744 tid 2 2024-09-17T16:02:14.414 INFO:tasks.rados.rados.0.smithi181.stdout:248: writing smithi18136759-18 from 2719744 to 3342336 tid 3 2024-09-17T16:02:14.414 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 34 current snap is 26 2024-09-17T16:02:14.414 INFO:tasks.rados.rados.0.smithi181.stdout:249: seq_num 105 ranges {1769472=606208,2375680=688128,3063808=475136} 2024-09-17T16:02:14.426 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:14.424+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:14.427 INFO:tasks.rados.rados.0.smithi181.stdout:249: writing smithi18136759-34 from 1769472 to 2375680 tid 1 2024-09-17T16:02:14.431 INFO:tasks.rados.rados.0.smithi181.stdout:249: writing smithi18136759-34 from 2375680 to 3063808 tid 2 2024-09-17T16:02:14.433 INFO:tasks.rados.rados.0.smithi181.stdout:249: writing smithi18136759-34 from 3063808 to 3538944 tid 3 2024-09-17T16:02:14.434 INFO:tasks.rados.rados.0.smithi181.stdout:250: snap_remove snap 12 2024-09-17T16:02:14.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.703271637, 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19491629299999999, 2024-09-17T16:02:14.455 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.456 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748958+0000", 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.457 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748966+0000", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5870000000000004e-06 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748979+0000", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748988+0000", 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.458 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749052+0000", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4618999999999999e-05 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749064+0000", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.939947+0000", 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19088328399999999 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.459 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940079+0000", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013157500000000001 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940245+0000", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016577900000000001 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943345+0000", 2024-09-17T16:02:14.460 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031004499999999998 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943806+0000", 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046112200000000001 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943843+0000", 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6578000000000001e-05 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.461 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943873+0000", 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [cmpxattr striper.excl (25) op 1 mode 1 in=12b,setxattr striper.excl in=12b,call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.504186310999998, 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83632086699999997, 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.462 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.463 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948043+0000", 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948048+0000", 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.464 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948060+0000", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1761e-05 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948066+0000", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5350000000000001e-06 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948111+0000", 2024-09-17T16:02:14.465 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4783000000000001e-05 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948190+0000", 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948399+0000", 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000209292 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948922+0000", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052251800000000003 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784264+0000", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83534246599999995 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784309+0000", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4660000000000003e-05 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.467 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784363+0000", 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4095000000000003e-05 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.4219.0:8 1.0 e24/23 ondisk, result = 0)", 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 57.503064115999997, 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83521008600000002, 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.468 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949165+0000", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949169+0000", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9339999999999999e-06 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784135+0000", 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83496651200000005 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784151+0000", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784250+0000", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8222999999999998e-05 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784260+0000", 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0265000000000001e-05 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784374+0000", 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114373 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4296.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 55.692875903000001, 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.241196405, 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4296", 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.6:0/1428345084", 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759356+0000", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.484e-06 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759366+0000", 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9089999999999995e-06 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.474 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759375+0000", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759397+0000", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1824e-05 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759444+0000", 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6539000000000003e-05 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759449+0000", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998167+0000", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23871730099999999 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998242+0000", 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4892000000000001e-05 2024-09-17T16:02:14.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0", 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998303+0000", 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0931999999999999e-05 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998708+0000", 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040511399999999998 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:14.477 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000499+0000", 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017911909999999999 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000526+0000", 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6798999999999999e-05 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000549+0000", 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3110000000000001e-05 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.478 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:34 3.2s0 3:41aa4422:::smithi18136759-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.595613399000001, 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015312129000000001, 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 34 2024-09-17T16:02:14.479 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.480 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856616+0000", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4899999999999999e-06 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856732+0000", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011513 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856745+0000", 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3677e-05 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.481 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856753+0000", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856824+0000", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1291999999999994e-05 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856962+0000", 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013730799999999999 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.482 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.857708+0000", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074624700000000001 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861225+0000", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035171249999999999 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871848+0000", 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010623323 2024-09-17T16:02:14.483 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871927+0000", 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8844000000000006e-05 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:35 3.2s0 3:41aa4422:::smithi18136759-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.595457859, 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015326866, 2024-09-17T16:02:14.484 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 35 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.485 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856771+0000", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856775+0000", 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.33e-06 2024-09-17T16:02:14.486 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856782+0000", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0800000000000002e-06 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856786+0000", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5759999999999999e-06 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858308+0000", 2024-09-17T16:02:14.487 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015218009999999999 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858393+0000", 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5502999999999998e-05 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859025+0000", 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000632009 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.488 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861267+0000", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022417639999999998 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872026+0000", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010758772999999999 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872097+0000", 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1390000000000006e-05 2024-09-17T16:02:14.489 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:36 3.2s0 3:41aa4422:::smithi18136759-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.595372515999998, 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017199677, 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.490 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 36 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.491 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856861+0000", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0759999999999996e-06 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856869+0000", 2024-09-17T16:02:14.492 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6319999999999997e-06 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856876+0000", 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5989999999999998e-06 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859529+0000", 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026531060000000001 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.493 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859628+0000", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9104000000000005e-05 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872415+0000", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012787056 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872500+0000", 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999998e-05 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.494 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874055+0000", 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015554810000000001 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=11, reqid=client.4560.0:50, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.581288850999996, 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016547899000000001, 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.495 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.496 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870940+0000", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870953+0000", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2798e-05 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881713+0000", 2024-09-17T16:02:14.497 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010760202 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881721+0000", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8949999999999998e-06 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881765+0000", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3778e-05 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881775+0000", 2024-09-17T16:02:14.498 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0135999999999999e-05 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887451+0000", 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056764090000000003 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887487+0000", 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6065000000000001e-05 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.499 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:81 3.2s0 3:4c2fa52d:::smithi18136759-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.558185051999999, 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015999928, 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.500 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 81 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.501 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894044+0000", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894045+0000", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065e-06 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894048+0000", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.489e-06 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.502 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894051+0000", 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9799999999999998e-06 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896400+0000", 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023486919999999999 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896473+0000", 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3380000000000006e-05 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.503 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908531+0000", 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012058016 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908615+0000", 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3968000000000007e-05 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.504 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.910043+0000", 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014280569999999999 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s3 35/32 ECSubWrite(tid=34, reqid=client.4560.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.526829200999998, 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021043124999999999, 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.505 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.506 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925400+0000", 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925410+0000", 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0345e-05 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944975+0000", 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019565013999999999 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.507 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944985+0000", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0257e-05 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945787+0000", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00080139199999999995 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945797+0000", 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9590000000000007e-06 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.508 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946410+0000", 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061359000000000001 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946442+0000", 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.509 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=42, reqid=client.4560.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 40.496471176999997, 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022376667999999999, 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.510 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955758+0000", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955765+0000", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.345e-06 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.511 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977438+0000", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021672416999999999 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977447+0000", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977511+0000", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4066000000000006e-05 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.512 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977519+0000", 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8329999999999994e-06 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978105+0000", 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058612099999999999 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978134+0000", 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8558e-05 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.513 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:268 3.2s0 3:406bd9c9:::smithi18136759-23:head [append 0~720896] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.528904673, 2024-09-17T16:02:14.514 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016657371000000001, 2024-09-17T16:02:14.515 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.515 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 268 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.516 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923326+0000", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0999999999999998e-06 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923806+0000", 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048034 2024-09-17T16:02:14.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923821+0000", 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5180999999999999e-05 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923831+0000", 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4450000000000002e-06 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.518 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923891+0000", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9858999999999997e-05 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923971+0000", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0493000000000003e-05 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.924982+0000", 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001010543 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929374+0000", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043921699999999999 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939921+0000", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010546754 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939981+0000", 2024-09-17T16:02:14.520 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0486000000000003e-05 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:271 3.2s0 3:406bd9c9:::smithi18136759-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.526268603999998, 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015374301, 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.521 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 271 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:14.522 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925961+0000", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925964+0000", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5319999999999998e-06 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925976+0000", 2024-09-17T16:02:14.523 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1795e-05 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925983+0000", 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.289e-06 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929232+0000", 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032491070000000002 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.524 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929352+0000", 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119639 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940555+0000", 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011203489000000001 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.525 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940634+0000", 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8165999999999997e-05 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.941334+0000", 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000700496 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:392 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~589824] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:14.526 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.109751974000002, 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018219264999999998, 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 392 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:14.527 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342479+0000", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.481e-06 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343310+0000", 2024-09-17T16:02:14.528 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083096699999999995 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343328+0000", 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8059000000000001e-05 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343340+0000", 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1664e-05 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.529 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343445+0000", 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010582900000000001 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343453+0000", 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348803+0000", 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053503420000000001 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348975+0000", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017183899999999999 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350509+0000", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015333320000000001 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356400+0000", 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058917320000000002 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360612+0000", 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042120680000000002 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360696+0000", 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:393 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~573440] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.105994945999999, 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.014645886, 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 393 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.533 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346233+0000", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346236+0000", 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2409999999999998e-06 2024-09-17T16:02:14.534 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346934+0000", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069824199999999996 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346952+0000", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7728999999999999e-05 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.346961+0000", 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0159999999999997e-06 2024-09-17T16:02:14.535 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347031+0000", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0168999999999999e-05 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347039+0000", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2190000000000004e-06 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.351923+0000", 2024-09-17T16:02:14.536 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048838550000000003 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352046+0000", 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012300599999999999 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352744+0000", 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00069777000000000001 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.537 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356401+0000", 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0036577049999999998 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360814+0000", 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044128459999999998 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360879+0000", 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5087999999999998e-05 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.538 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:396 3.2s0 3:465e1f11:::smithi18136759-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 16.104490567999999, 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015698437999999999, 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 396 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.539 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.540 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8699999999999995e-07 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347741+0000", 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6690000000000002e-06 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347747+0000", 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.455e-06 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.541 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347752+0000", 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0459999999999996e-06 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356238+0000", 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084855650000000005 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.542 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356325+0000", 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7039999999999993e-05 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361673+0000", 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053483849999999998 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361762+0000", 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8825000000000004e-05 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.543 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.363436+0000", 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001673966 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:484 3.18s0 3:19aefe3f:::smithi18136759-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.1012704680000001, 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017624631000000002, 2024-09-17T16:02:14.544 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 484 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.545 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350961+0000", 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:02:14.546 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352262+0000", 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001300708 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352278+0000", 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5848000000000001e-05 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.547 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354256+0000", 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019783330000000001 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354327+0000", 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0839000000000004e-05 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363568+0000", 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092410559999999992 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.548 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363668+0000", 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9382000000000005e-05 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368582+0000", 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049148810000000003 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.549 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:488 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [setxattr _header (57) in=64b] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.0881085329999998, 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83388352600000004, 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 488 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.550 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.551 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364123+0000", 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9220000000000001e-06 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364131+0000", 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.552 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364137+0000", 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4679999999999997e-06 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366664+0000", 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025266949999999998 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366744+0000", 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0739000000000001e-05 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.553 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367168+0000", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042360999999999999 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368066+0000", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089781599999999996 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197852+0000", 2024-09-17T16:02:14.554 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82978574900000002 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198003+0000", 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015172699999999999 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:489 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:14.555 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.0880823130000001, 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83494334199999998, 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 489 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.556 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364147+0000", 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.557 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364149+0000", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2359999999999999e-06 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364156+0000", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4630000000000004e-06 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364163+0000", 2024-09-17T16:02:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368609+0000", 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044454869999999997 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368658+0000", 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9438999999999999e-05 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.559 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198316+0000", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965781999999999 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198436+0000", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012003299999999999 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.199089+0000", 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000653445 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.560 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.18s0 69/53 ECSubWriteReply(tid=224, last_complete=69'42, committed=1, applied=1))", 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.0843035959999998, 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83034511799999999, 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.561 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.13e-07 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.562 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367929+0000", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197585+0000", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965586700000005 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197599+0000", 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4369000000000001e-05 2024-09-17T16:02:14.563 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197711+0000", 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011220199999999999 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198270+0000", 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00055871000000000002 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.564 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:14.567 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:14.564+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:14.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:14.666 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:14.664+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:14.670 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:14.670 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:14.670 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.679 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:14.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:02:14.767 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:14.764+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:14.802 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:14.802 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:14.823 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:14.823 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:14.823 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:14.824 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:14.824 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:14.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:14.847 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:14.847 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:14.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:02:14.868 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:14.866+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:14.869 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:14.869 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:14.888 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:14.889 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:14.912 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:14.913 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:14.934 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:14.934 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:14.955 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:14.955 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:14.968 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:14.966+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.9s3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.976 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.113329411000002, 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053801109999999999, 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.977 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860215+0000", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860418+0000", 2024-09-17T16:02:14.978 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00020234499999999999 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860448+0000", 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0692999999999999e-05 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860458+0000", 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6460000000000005e-06 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.979 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860528+0000", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0272999999999996e-05 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.860536+0000", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1040000000000007e-06 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865550+0000", 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050133510000000001 2024-09-17T16:02:14.980 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.865595+0000", 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5396999999999999e-05 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=4, reqid=client.4560.0:5, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:14.981 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.109293686999997, 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051579750000000004, 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.982 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864251+0000", 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864260+0000", 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5769999999999994e-06 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.983 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864296+0000", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6227999999999999e-05 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864304+0000", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0090000000000007e-06 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864392+0000", 2024-09-17T16:02:14.984 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8549999999999998e-05 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.864402+0000", 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3279999999999995e-06 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869377+0000", 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.004975045 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.985 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.869409+0000", 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1953999999999999e-05 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.106081271999997, 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.986 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.987 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:02:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:02:14.989 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.990 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.105691559999997, 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.991 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.992 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.993 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.994 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.096271151000003, 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:14.995 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:14.996 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:14.997 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:02:14.999 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.041585626, 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.000 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.001 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:02:15.002 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:02:15.003 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:02:15.004 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.041203226999997, 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:02:15.005 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.006 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.007 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.008 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.009 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.010 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.041167434000002, 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.011 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.012 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.013 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.014 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:02:15.015 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.041139268999999, 2024-09-17T16:02:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.017 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:02:15.018 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.019 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:02:15.020 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.021 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.040865750000002, 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.022 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.023 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:02:15.024 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:02:15.025 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.026 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.027 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.040854693999997, 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:02:15.028 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:15.029 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.030 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.031 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:02:15.032 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.008099541, 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:02:15.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:15.039 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.007610491000001, 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:02:15.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:02:15.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.043 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 41.007569641000003, 2024-09-17T16:02:15.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:02:15.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:02:15.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.050 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.059850045000001, 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073441670000000004, 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913695+0000", 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913725+0000", 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9743999999999998e-05 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919758+0000", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00603318 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919766+0000", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3610000000000001e-06 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919835+0000", 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9336000000000002e-05 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919844+0000", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921001+0000", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115653 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921039+0000", 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7700000000000002e-05 2024-09-17T16:02:15.054 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 46/32 ECSubWriteReply(tid=17, last_complete=46'12, committed=1, applied=1))", 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 30.049863468000002, 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.330502611, 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.055 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923682+0000", 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:02:15.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923686+0000", 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253962+0000", 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33027621400000001 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253975+0000", 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254119+0000", 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014402100000000001 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254184+0000", 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4158999999999993e-05 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.058 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:346 3.3s0 3:c1daf1ed:::smithi18136759-38:head [delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.614774647000001, 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069854690000000002, 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358772+0000", 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0810000000000001e-06 2024-09-17T16:02:15.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358779+0000", 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2849999999999997e-06 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358790+0000", 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9759999999999995e-06 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358838+0000", 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9730000000000001e-05 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358966+0000", 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012749199999999999 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.359434+0000", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046890000000000001 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.362289+0000", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028544030000000001 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365699+0000", 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034100210000000001 2024-09-17T16:02:15.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365755+0000", 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6456000000000003e-05 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.10s3 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.627403207, 2024-09-17T16:02:15.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081444710000000003, 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:15.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-07 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346159+0000", 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7445e-05 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346307+0000", 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014812199999999999 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346314+0000", 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.477e-06 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346436+0000", 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012286999999999999 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346445+0000", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.833e-06 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354242+0000", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077967399999999999 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354286+0000", 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3801999999999998e-05 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.16s2 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 12.625071386, 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059176979999999999, 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348474+0000", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5799999999999999e-07 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348485+0000", 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0892e-05 2024-09-17T16:02:15.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348510+0000", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5287999999999999e-05 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348517+0000", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.267e-06 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348561+0000", 2024-09-17T16:02:15.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4360000000000002e-05 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348568+0000", 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7270000000000001e-06 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354355+0000", 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057869480000000001 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354391+0000", 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5658000000000003e-05 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.18s3 69/53 ECSubWrite(tid=217, reqid=client.4560.0:481, at_version=69'35, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 7.6231625940000001, 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0050637679999999997, 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:15.073 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350382+0000", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350383+0000", 2024-09-17T16:02:15.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350488+0000", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010493999999999999 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350525+0000", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7608000000000003e-05 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350534+0000", 2024-09-17T16:02:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350610+0000", 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6055000000000003e-05 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350621+0000", 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0485999999999999e-05 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:15.076 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.355408+0000", 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047872100000000001 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.355446+0000", 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7858999999999998e-05 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:15.077 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:15.078 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:15.079 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:15.079 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:15.080 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:15.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:15.080 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:15.081 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:15.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2024-09-17T16:02:15.082 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:15.082 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:15.082 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:15.083 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:15.083 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:15.077+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:15.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:15.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:15.090 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:15.088+0000 7fd6a7fff640 0 -- 172.21.15.6:0/1835715554 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fd6a8221bf0 msgr2=0x7fd6a8223fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:15.112 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:15.112 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:15.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:15.180 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:15.178+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:15.193 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.193 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:15.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 reset_purged_snaps_last 2024-09-17T16:02:15.281 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:15.279+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:15.311 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.311 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:15.382 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:15.381+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:15.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:02:15.407 INFO:tasks.rados.rados.0.smithi181.stdout:250: done (2 left) 2024-09-17T16:02:15.407 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 20 current snap is 26 2024-09-17T16:02:15.407 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 1 to smithi18136759-18 2024-09-17T16:02:15.407 INFO:tasks.rados.rados.0.smithi181.stdout:251: seq_num 106 ranges {0=622592,622592=442368,1064960=524288,1589248=196608} 2024-09-17T16:02:15.410 INFO:tasks.rados.rados.0.smithi181.stdout:251: writing smithi18136759-20 from 0 to 622592 tid 1 2024-09-17T16:02:15.414 INFO:tasks.rados.rados.0.smithi181.stdout:251: writing smithi18136759-20 from 622592 to 1064960 tid 2 2024-09-17T16:02:15.418 INFO:tasks.rados.rados.0.smithi181.stdout:251: writing smithi18136759-20 from 1064960 to 1589248 tid 3 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:251: writing smithi18136759-20 from 1589248 to 1785856 tid 4 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:252: setattr oid 9 current snap is 26 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 2 to smithi18136759-18 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 3 to smithi18136759-18 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 4 to smithi18136759-18 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: finishing write tid 5 to smithi18136759-18 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 18 updating version 0 to 135 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 18 version 135 is already newer than 133 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 18 version 135 is already newer than 132 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:248: oid 18 version 135 is already newer than 134 2024-09-17T16:02:15.420 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 135 (ObjNum 104 snap 26 seq_num 104) dirty exists 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:248: left oid 18 (ObjNum 104 snap 26 seq_num 104) 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing write tid 1 to smithi18136759-34 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing write tid 2 to smithi18136759-34 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing write tid 3 to smithi18136759-34 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing write tid 4 to smithi18136759-34 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: finishing write tid 5 to smithi18136759-34 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: oid 34 updating version 0 to 18 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: oid 34 version 18 is already newer than 15 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: oid 34 version 18 is already newer than 16 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: oid 34 version 18 is already newer than 17 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 18 (ObjNum 105 snap 26 seq_num 105) dirty exists 2024-09-17T16:02:15.421 INFO:tasks.rados.rados.0.smithi181.stdout:249: left oid 34 (ObjNum 105 snap 26 seq_num 105) 2024-09-17T16:02:15.422 INFO:tasks.rados.rados.0.smithi181.stdout:248: done (3 left) 2024-09-17T16:02:15.422 INFO:tasks.rados.rados.0.smithi181.stdout:249: done (2 left) 2024-09-17T16:02:15.422 INFO:tasks.rados.rados.0.smithi181.stdout:253: copy_from oid 19 from oid 46 current snap is 26 2024-09-17T16:02:15.422 INFO:tasks.rados.rados.0.smithi181.stdout:254: delete oid 31 current snap is 26 2024-09-17T16:02:15.422 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 1 to smithi18136759-20 2024-09-17T16:02:15.424 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 2 to smithi18136759-20 2024-09-17T16:02:15.428 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 3 to smithi18136759-20 2024-09-17T16:02:15.429 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 4 to smithi18136759-20 2024-09-17T16:02:15.429 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 5 to smithi18136759-20 2024-09-17T16:02:15.431 INFO:tasks.rados.rados.0.smithi181.stdout:251: finishing write tid 6 to smithi18136759-20 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: oid 20 updating version 0 to 47 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: oid 20 version 47 is already newer than 46 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: oid 20 updating version 47 to 48 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: oid 20 version 48 is already newer than 44 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: oid 20 version 48 is already newer than 45 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 20 v 48 (ObjNum 106 snap 26 seq_num 106) dirty exists 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:251: left oid 20 (ObjNum 106 snap 26 seq_num 106) 2024-09-17T16:02:15.432 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 43 (ObjNum 289 snap 0 seq_num 0) dirty exists 2024-09-17T16:02:15.437 INFO:tasks.rados.rados.0.smithi181.stdout:253: finishing copy_from racing read to smithi18136759-19 2024-09-17T16:02:15.439 INFO:tasks.rados.rados.0.smithi181.stdout:253: finishing copy_from to smithi18136759-19 2024-09-17T16:02:15.439 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-17T16:02:15.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2024-09-17T16:02:15.484 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:15.482+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:15.490 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.490 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:15.585 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:15.582+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:15.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2024-09-17T16:02:15.654 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.654 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:15.686 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:15.683+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:15.787 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:15.785+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:15.815 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.815 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:15.888 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:15.886+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:15.970 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:15.971 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:15.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:15.989 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:15.987+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:16.086 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.086 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.089 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:16.087+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:16.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:16.190 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:16.188+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:16.202 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.202 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:16.214 INFO:tasks.rados.rados.0.smithi181.stdout:251: done (3 left) 2024-09-17T16:02:16.215 INFO:tasks.rados.rados.0.smithi181.stdout:252: done (2 left) 2024-09-17T16:02:16.215 INFO:tasks.rados.rados.0.smithi181.stdout:253: done (1 left) 2024-09-17T16:02:16.215 INFO:tasks.rados.rados.0.smithi181.stdout:254: done (0 left) 2024-09-17T16:02:16.215 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 1 current snap is 26 2024-09-17T16:02:16.215 INFO:tasks.rados.rados.0.smithi181.stdout:255: seq_num 108 ranges {1507328=655360,2162688=606208,2768896=475136,3244032=81920} 2024-09-17T16:02:16.226 INFO:tasks.rados.rados.0.smithi181.stdout:255: writing smithi18136759-1 from 1507328 to 2162688 tid 1 2024-09-17T16:02:16.229 INFO:tasks.rados.rados.0.smithi181.stdout:255: writing smithi18136759-1 from 2162688 to 2768896 tid 2 2024-09-17T16:02:16.232 INFO:tasks.rados.rados.0.smithi181.stdout:255: writing smithi18136759-1 from 2768896 to 3244032 tid 3 2024-09-17T16:02:16.233 INFO:tasks.rados.rados.0.smithi181.stdout:255: writing smithi18136759-1 from 3244032 to 3325952 tid 4 2024-09-17T16:02:16.233 INFO:tasks.rados.rados.0.smithi181.stdout:256: snap_remove snap 24 2024-09-17T16:02:16.291 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:16.289+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:16.358 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.358 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:16.391 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:16.389+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:02:16.407 INFO:tasks.rados.rados.0.smithi181.stdout:256: done (1 left) 2024-09-17T16:02:16.408 INFO:tasks.rados.rados.0.smithi181.stdout:257: snap_remove snap 25 2024-09-17T16:02:16.476 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.477 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:16.492 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:16.490+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:16.593 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.593 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.594 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:16.592+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:16.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:16.694 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:16.692+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:16.709 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.709 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:16.795 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:16.793+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:16.840 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.840 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:16.897 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:16.895+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:16.958 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:16.958 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:16.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:16.998 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:16.996+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:17.074 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:17.074 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:17.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:17.099 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:17.096+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:17.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:17.190 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:17.191 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:17.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:17.200 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:17.197+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:17.259 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:17.259 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:17.281 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:17.281 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:17.300 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:17.297+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:17.303 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:17.303 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:17.325 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:17.325 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:17.346 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:17.346 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:17.346 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:17.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:17.367 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:17.367 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:17.388 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:17.389 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:17.400 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:17.398+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:17.409 INFO:tasks.rados.rados.0.smithi181.stdout:257: done (1 left) 2024-09-17T16:02:17.409 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 10 current snap is 26 2024-09-17T16:02:17.409 INFO:tasks.rados.rados.0.smithi181.stdout:258: seq_num 109 ranges {1900544=671744,2572288=753664,3325952=409600} 2024-09-17T16:02:17.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:17.411 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:17.423 INFO:tasks.rados.rados.0.smithi181.stdout:258: writing smithi18136759-10 from 1900544 to 2572288 tid 1 2024-09-17T16:02:17.426 INFO:tasks.rados.rados.0.smithi181.stdout:258: writing smithi18136759-10 from 2572288 to 3325952 tid 2 2024-09-17T16:02:17.429 INFO:tasks.rados.rados.0.smithi181.stdout:258: writing smithi18136759-10 from 3325952 to 3735552 tid 3 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:259: rollback oid 26 current snap is 26 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 26 to 4 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 1 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 2 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 3 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 4 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 5 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: finishing write tid 6 to smithi18136759-1 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: oid 1 updating version 0 to 55 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: oid 1 version 55 is already newer than 52 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: oid 1 version 55 is already newer than 53 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: oid 1 version 55 is already newer than 54 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:255: oid 1 updating version 55 to 56 2024-09-17T16:02:17.430 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 56 (ObjNum 108 snap 26 seq_num 108) dirty exists 2024-09-17T16:02:17.431 INFO:tasks.rados.rados.0.smithi181.stdout:255: left oid 1 (ObjNum 108 snap 26 seq_num 108) 2024-09-17T16:02:17.431 INFO:tasks.rados.rados.0.smithi181.stdout:255: done (2 left) 2024-09-17T16:02:17.431 INFO:tasks.rados.rados.0.smithi181.stdout:260: delete oid 50 current snap is 26 2024-09-17T16:02:17.435 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing write tid 1 to smithi18136759-10 2024-09-17T16:02:17.438 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing write tid 2 to smithi18136759-10 2024-09-17T16:02:17.438 INFO:tasks.rados.rados.0.smithi181.stdout:259: finishing rollback tid 0 to smithi18136759-26 2024-09-17T16:02:17.439 INFO:tasks.rados.rados.0.smithi181.stdout:259: finishing rollback tid 1 to smithi18136759-26 2024-09-17T16:02:17.440 INFO:tasks.rados.rados.0.smithi181.stdout:259: finishing rollback tid 2 to smithi18136759-26 2024-09-17T16:02:17.440 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 26 v 28 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:259: done (2 left) 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:260: done (1 left) 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:261: copy_from oid 2 from oid 50 current snap is 26 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:262: rollback oid 48 current snap is 26 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 48 to 10 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:263: read oid 32 snap 23 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:263: expect deleted 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:264: rollback oid 9 current snap is 26 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 9 to 20 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 16 current snap is 26 2024-09-17T16:02:17.441 INFO:tasks.rados.rados.0.smithi181.stdout:265: seq_num 111 ranges {0=704512,704512=655360,1359872=507904} 2024-09-17T16:02:17.442 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:17.442 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:17.444 INFO:tasks.rados.rados.0.smithi181.stdout:265: writing smithi18136759-16 from 0 to 704512 tid 1 2024-09-17T16:02:17.448 INFO:tasks.rados.rados.0.smithi181.stdout:265: writing smithi18136759-16 from 704512 to 1359872 tid 2 2024-09-17T16:02:17.452 INFO:tasks.rados.rados.0.smithi181.stdout:265: writing smithi18136759-16 from 1359872 to 1867776 tid 3 2024-09-17T16:02:17.452 INFO:tasks.rados.rados.0.smithi181.stdout:266: read oid 20 snap -1 2024-09-17T16:02:17.452 INFO:tasks.rados.rados.0.smithi181.stdout:266: expect (ObjNum 106 snap 26 seq_num 106) 2024-09-17T16:02:17.452 INFO:tasks.rados.rados.0.smithi181.stdout:267: read oid 36 snap -1 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:267: expect deleted 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:268: read oid 5 snap 14 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:268: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:269: setattr oid 12 current snap is 26 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing write tid 3 to smithi18136759-10 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing write tid 4 to smithi18136759-10 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: finishing write tid 5 to smithi18136759-10 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: oid 10 updating version 0 to 152 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: oid 10 updating version 152 to 154 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: oid 10 version 154 is already newer than 153 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: oid 10 version 154 is already newer than 149 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 154 (ObjNum 109 snap 26 seq_num 109) dirty exists 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:258: left oid 10 (ObjNum 109 snap 26 seq_num 109) 2024-09-17T16:02:17.453 INFO:tasks.rados.rados.0.smithi181.stdout:264: finishing rollback tid 0 to smithi18136759-9 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:264: finishing rollback tid 1 to smithi18136759-9 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 45 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing rollback tid 0 to smithi18136759-48 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:261: finishing copy_from to smithi18136759-2 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:261: got expected ENOENT (src dne) 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:261: finishing copy_from racing read to smithi18136759-2 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing rollback tid 1 to smithi18136759-48 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:262: finishing rollback tid 2 to smithi18136759-48 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 81 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:265: finishing write tid 1 to smithi18136759-16 2024-09-17T16:02:17.454 INFO:tasks.rados.rados.0.smithi181.stdout:265: finishing write tid 2 to smithi18136759-16 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:258: done (9 left) 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:261: done (8 left) 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:262: done (7 left) 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:264: done (6 left) 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:267: done (5 left) 2024-09-17T16:02:17.455 INFO:tasks.rados.rados.0.smithi181.stdout:270: snap_create 2024-09-17T16:02:17.456 INFO:tasks.rados.rados.0.smithi181.stdout:265: finishing write tid 3 to smithi18136759-16 2024-09-17T16:02:17.458 INFO:tasks.rados.rados.0.smithi181.stdout:265: finishing write tid 4 to smithi18136759-16 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 12 v 47 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: finishing write tid 5 to smithi18136759-16 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: oid 16 updating version 0 to 29 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: oid 16 version 29 is already newer than 27 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: oid 16 version 29 is already newer than 28 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: oid 16 updating version 29 to 30 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 30 (ObjNum 111 snap 26 seq_num 111) dirty exists 2024-09-17T16:02:17.459 INFO:tasks.rados.rados.0.smithi181.stdout:265: left oid 16 (ObjNum 111 snap 26 seq_num 111) 2024-09-17T16:02:17.460 INFO:tasks.rados.rados.0.smithi181.stdout:268: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-17T16:02:17.461 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 45073 2024-09-17T16:02:17.461 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 34319 2024-09-17T16:02:17.462 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 27601 2024-09-17T16:02:17.462 INFO:tasks.rados.rados.0.smithi181.stdout:266: expect (ObjNum 106 snap 26 seq_num 106) 2024-09-17T16:02:17.462 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:17.466 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:17.466 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:17.476 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:17.476 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:17.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.545284737999999, 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:02:17.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=35, reqid=client.4560.0:242, at_version=44'12, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:02:17.481 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.543581320000001, 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023531030000000001, 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930338+0000", 2024-09-17T16:02:17.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.930339+0000", 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1909999999999999e-06 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931084+0000", 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074480300000000004 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931122+0000", 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8195000000000003e-05 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931132+0000", 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0499000000000001e-05 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931291+0000", 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015850099999999999 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931302+0000", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1487e-05 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932657+0000", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001355001 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932691+0000", 2024-09-17T16:02:17.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3426000000000002e-05 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.554615747, 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026452619999999998, 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.14e-07 2024-09-17T16:02:17.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919991+0000", 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068760399999999997 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920025+0000", 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4292000000000003e-05 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920043+0000", 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8028000000000001e-05 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920118+0000", 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4583000000000001e-05 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920129+0000", 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1105e-05 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921921+0000", 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017915590000000001 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921949+0000", 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7977e-05 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.490 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.548532922, 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:02:17.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.541707305999999, 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.495 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:02:17.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.122382898000001, 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017347922000000002, 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351537+0000", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351558+0000", 2024-09-17T16:02:17.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1399e-05 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367847+0000", 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289428000000002 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367858+0000", 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0835e-05 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367938+0000", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0104000000000004e-05 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367950+0000", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1153e-05 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368846+0000", 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089654899999999996 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368884+0000", 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8075999999999999e-05 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.123672687999999, 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020622370000000002, 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350247+0000", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.67e-07 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350271+0000", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3926000000000001e-05 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350316+0000", 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5837000000000002e-05 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350327+0000", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0597e-05 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350399+0000", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2203e-05 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.350412+0000", 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2364999999999999e-05 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.352262+0000", 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018508299999999999 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.352309+0000", 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6312e-05 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s4 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.101027017, 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026391370000000002, 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.372892+0000", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.29e-07 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373128+0000", 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023566099999999999 2024-09-17T16:02:17.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373174+0000", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6307999999999998e-05 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373186+0000", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2084e-05 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373282+0000", 2024-09-17T16:02:17.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5749999999999996e-05 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373296+0000", 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3468999999999999e-05 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375480+0000", 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002184348 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.375531+0000", 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1388000000000003e-05 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.08078879, 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047653560000000001, 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.513 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393131+0000", 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9600000000000008e-07 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394022+0000", 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089089400000000004 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394066+0000", 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4193000000000002e-05 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394089+0000", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2608000000000001e-05 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394165+0000", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5638999999999999e-05 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394175+0000", 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0277e-05 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397846+0000", 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036707889999999998 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397896+0000", 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9960000000000003e-05 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.517 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.077280002999998, 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020037330000000002, 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396639+0000", 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.29e-07 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396866+0000", 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022683499999999999 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396945+0000", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8825999999999994e-05 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396956+0000", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1301000000000001e-05 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397067+0000", 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110486 2024-09-17T16:02:17.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397078+0000", 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1501999999999999e-05 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398596+0000", 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015176580000000001 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398643+0000", 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6896e-05 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=64'34, trim_to=0'0, roll_forward_to=62'32))", 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.136638365, 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035966639999999999, 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.522 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:17.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.04e-07 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337334+0000", 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3352e-05 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337390+0000", 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5319999999999999e-05 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337400+0000", 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337509+0000", 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108571 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337523+0000", 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4211e-05 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340822+0000", 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032988689999999998 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340877+0000", 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5481e-05 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 64/32 ECSubWrite(tid=128, reqid=osd.12.0:128, at_version=64'68, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:17.526 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.337725+0000", 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.136194041, 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023396670000000001, 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337725+0000", 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337725+0000", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337725+0000", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4999999999999999e-07 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337750+0000", 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4813000000000001e-05 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337788+0000", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8095e-05 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337799+0000", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.062e-05 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337881+0000", 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2130000000000001e-05 2024-09-17T16:02:17.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337893+0000", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340014+0000", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021203260000000001 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340065+0000", 2024-09-17T16:02:17.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1023000000000003e-05 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.134628595000001, 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027044159999999999, 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.531 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339292+0000", 2024-09-17T16:02:17.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.187e-06 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339317+0000", 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5355e-05 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339357+0000", 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0244000000000003e-05 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339368+0000", 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0512e-05 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339491+0000", 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012284700000000001 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339502+0000", 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0848e-05 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341950+0000", 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002448703 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341995+0000", 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4719999999999999e-05 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.535 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=42, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=64'34))", 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.128858112, 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027339, 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.28e-07 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345069+0000", 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6920000000000002e-06 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345381+0000", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031179599999999998 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345390+0000", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5810000000000003e-06 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345492+0000", 2024-09-17T16:02:17.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101902 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345506+0000", 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3395e-05 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347746+0000", 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002240845 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347795+0000", 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8461000000000002e-05 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s2 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.128606638999999, 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035907539999999998, 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.540 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345313+0000", 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345362+0000", 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8804000000000002e-05 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345410+0000", 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8133999999999997e-05 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345419+0000", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6039999999999999e-06 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345499+0000", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0173000000000006e-05 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345510+0000", 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348833+0000", 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033223749999999998 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348903+0000", 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0499000000000004e-05 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s4 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345753+0000", 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.128166211, 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021140759999999999, 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345753+0000", 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345753+0000", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345753+0000", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.99e-07 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345780+0000", 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7100000000000001e-05 2024-09-17T16:02:17.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345825+0000", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4836000000000002e-05 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345833+0000", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2130000000000008e-06 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345937+0000", 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104115 2024-09-17T16:02:17.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345948+0000", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0954999999999999e-05 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347818+0000", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018699750000000001 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347867+0000", 2024-09-17T16:02:17.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8683000000000001e-05 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.127209354, 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034797959999999998, 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.549 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346711+0000", 2024-09-17T16:02:17.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6600000000000005e-07 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346735+0000", 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4652e-05 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346773+0000", 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7855999999999998e-05 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346782+0000", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5599999999999994e-06 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346854+0000", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2512e-05 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346865+0000", 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0957e-05 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350146+0000", 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032811709999999998 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350190+0000", 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3222000000000001e-05 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.553 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.350677+0000", 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.123241766, 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002380957, 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350677+0000", 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350677+0000", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350678+0000", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7400000000000003e-07 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350704+0000", 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6157e-05 2024-09-17T16:02:17.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350743+0000", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8603999999999998e-05 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350756+0000", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2938e-05 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350865+0000", 2024-09-17T16:02:17.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010969200000000001 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350877+0000", 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.136e-05 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353002+0000", 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021256069999999998 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353058+0000", 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5924999999999997e-05 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 76/32 ECSubWrite(tid=207, reqid=client.4560.0:597, at_version=76'132, trim_to=0'0, roll_forward_to=75'130))", 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.0647735919999999, 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045991310000000002, 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410426+0000", 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012808170000000001 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410463+0000", 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6449000000000003e-05 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410471+0000", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8420000000000005e-06 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410556+0000", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5426999999999994e-05 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410574+0000", 2024-09-17T16:02:17.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7306999999999999e-05 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413700+0000", 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031262460000000001 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413745+0000", 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4910000000000002e-05 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.11s4 76/65 ECSubWrite(tid=6, reqid=client.4560.0:603, at_version=76'16, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:14.438486+0000", 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.0354330310000002, 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020525539999999998, 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438486+0000", 2024-09-17T16:02:17.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438486+0000", 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438487+0000", 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1999999999999998e-07 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438929+0000", 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044234100000000001 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438962+0000", 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2540999999999999e-05 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.438968+0000", 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3770000000000001e-06 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.439033+0000", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5172999999999994e-05 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.439041+0000", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0450000000000001e-06 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.440495+0000", 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001453219 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.440539+0000", 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3937999999999998e-05 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.567 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:17.568 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:17.569 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:17.569 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:17.569 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:17.570 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:17.570 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:17.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:17.571 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:17.566+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:17.571 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:17.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:17.571 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:17.572 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:17.572 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:17.553+0000 7f8eebfff640 0 -- 172.21.15.6:0/3514948409 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f8eec221a40 msgr2=0x7f8eec223e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:17.669 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:17.667+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:17.718 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:17.718 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:17.718 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:17.718 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:17.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:17.770 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:17.768+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:17.871 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:17.869+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:17.874 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:17.874 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:17.874 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:17.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:17.972 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:17.971+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=0, reqid=client.4560.0:16, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.132195377000002, 2024-09-17T16:02:17.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001864189, 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856327+0000", 2024-09-17T16:02:17.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856328+0000", 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3570000000000001e-06 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856466+0000", 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000137415 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856516+0000", 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0541999999999998e-05 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856527+0000", 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0978000000000001e-05 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4474999999999994e-05 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856611+0000", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0720000000000006e-06 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858143+0000", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015325499999999999 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858191+0000", 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7800000000000003e-05 2024-09-17T16:02:17.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.131573938999999, 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:17.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:02:17.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:02:17.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:02:17.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.128611446999997, 2024-09-17T16:02:18.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:18.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:02:18.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:02:18.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.125036324, 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001910307, 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863486+0000", 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863487+0000", 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4900000000000004e-07 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863535+0000", 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7713999999999998e-05 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863653+0000", 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117757 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863662+0000", 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9779999999999994e-06 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863800+0000", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013886599999999999 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863813+0000", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.295e-05 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865357+0000", 2024-09-17T16:02:18.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001543464 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865396+0000", 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9629000000000003e-05 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:18.009 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.124981116999997, 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:18.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:02:18.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.013 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 35/32 ECSubWrite(tid=2, reqid=client.4560.0:61, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.103894293000003, 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018138760000000001, 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:02:18.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884628+0000", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9e-08 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.884974+0000", 2024-09-17T16:02:18.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034541900000000002 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885013+0000", 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9020000000000002e-05 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885023+0000", 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9000000000000001e-06 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885103+0000", 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9871999999999997e-05 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.885112+0000", 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4769999999999992e-06 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886400+0000", 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012878869999999999 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.886442+0000", 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2202e-05 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:02:18.018 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.100820292000002, 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:02:18.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:02:18.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.096268332000001, 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.023 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:02:18.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.027 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.061865392999998, 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019463340000000001, 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9200000000000002e-07 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037e-05 2024-09-17T16:02:18.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926741+0000", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0134999999999999e-05 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926751+0000", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926822+0000", 2024-09-17T16:02:18.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0667000000000006e-05 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926854+0000", 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928578+0000", 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017237979999999999 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928603+0000", 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5769e-05 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.070089623000001, 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020717940000000001, 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.032 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6899999999999999e-07 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918457+0000", 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918505+0000", 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7648000000000002e-05 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8609999999999993e-06 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918721+0000", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020663600000000001 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918732+0000", 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1058e-05 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920477+0000", 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017447840000000001 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920505+0000", 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7815000000000001e-05 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.036 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.064253720000004, 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020194549999999999, 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1600000000000002e-07 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924293+0000", 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924390+0000", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6893999999999998e-05 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924399+0000", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924465+0000", 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309000000000005e-05 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924474+0000", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5289999999999997e-06 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926235+0000", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001760529 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926288+0000", 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3263000000000001e-05 2024-09-17T16:02:18.040 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s6 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.615521266000002, 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003094585, 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.61e-07 2024-09-17T16:02:18.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373230+0000", 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022871100000000001 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373269+0000", 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8470999999999998e-05 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373278+0000", 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8449999999999998e-06 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373359+0000", 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0401999999999994e-05 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373368+0000", 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1919999999999994e-06 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376052+0000", 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026842319999999999 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376096+0000", 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3571e-05 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 24.591696282000001, 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020885859999999999, 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397062+0000", 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236006 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397106+0000", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3995000000000001e-05 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397119+0000", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397201+0000", 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2498999999999997e-05 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397214+0000", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2819999999999999e-05 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398870+0000", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016554829999999999 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398915+0000", 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5083000000000002e-05 2024-09-17T16:02:18.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.645852357000001, 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038094829999999998, 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342671+0000", 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342695+0000", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4366999999999999e-05 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343531+0000", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083620199999999997 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343542+0000", 2024-09-17T16:02:18.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0672e-05 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343828+0000", 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000285834 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343846+0000", 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8471999999999999e-05 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346440+0000", 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025931499999999998 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346480+0000", 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0064000000000001e-05 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:18.054 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.642782235, 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027362839999999999, 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1799999999999999e-07 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0065999999999999e-05 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345845+0000", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4876999999999996e-05 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345856+0000", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1073000000000001e-05 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345935+0000", 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9010000000000004e-05 2024-09-17T16:02:18.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345947+0000", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1708999999999999e-05 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348437+0000", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024899560000000002 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348477+0000", 2024-09-17T16:02:18.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9375000000000002e-05 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s5 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.642741611, 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034936559999999999, 2024-09-17T16:02:18.059 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.92e-07 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345815+0000", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3519999999999998e-05 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345868+0000", 2024-09-17T16:02:18.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3248000000000002e-05 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345877+0000", 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7970000000000006e-06 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346334+0000", 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045731999999999999 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346345+0000", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0854e-05 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349226+0000", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028807569999999998 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349275+0000", 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8968000000000003e-05 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.641890339, 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002590527, 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:18.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.73e-07 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346659+0000", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7055000000000001e-05 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346738+0000", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8388999999999998e-05 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346746+0000", 2024-09-17T16:02:18.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4090000000000003e-06 2024-09-17T16:02:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346817+0000", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0895999999999993e-05 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346826+0000", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349183+0000", 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023570639999999999 2024-09-17T16:02:18.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349223+0000", 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607999999999998e-05 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 15.641860612, 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037577729999999998, 2024-09-17T16:02:18.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346683+0000", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0866999999999999e-05 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347481+0000", 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079794800000000004 2024-09-17T16:02:18.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347491+0000", 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348761+0000", 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001270188 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348771+0000", 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.815e-06 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350377+0000", 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016056499999999999 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350420+0000", 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2944000000000001e-05 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.637558178000001, 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027021179999999999, 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350965+0000", 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.99e-07 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350989+0000", 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4145999999999999e-05 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351039+0000", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9858000000000003e-05 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351051+0000", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2729e-05 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351123+0000", 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1680000000000005e-05 2024-09-17T16:02:18.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351135+0000", 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2167000000000001e-05 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353616+0000", 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002481287 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353666+0000", 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0052000000000001e-05 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/53 ECSubWrite(tid=98, reqid=client.4560.0:500, at_version=69'48, trim_to=0'0, roll_forward_to=69'46))", 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.407886+0000", 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.580636705, 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018042069999999999, 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407886+0000", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407886+0000", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407887+0000", 2024-09-17T16:02:18.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4600000000000004e-07 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407911+0000", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4423e-05 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407946+0000", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5256999999999998e-05 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.407952+0000", 2024-09-17T16:02:18.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8540000000000002e-06 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.408001+0000", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9295999999999997e-05 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.408010+0000", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.064e-06 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.409656+0000", 2024-09-17T16:02:18.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001645259 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.409690+0000", 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4307999999999997e-05 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.083 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:18.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:18.085 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:18.083+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:18.186 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:18.184+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:18.193 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:18.193 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:18.193 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:18.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:18.287 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:18.285+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:18.349 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:18.349 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:18.349 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:18.349 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:18.349 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:18.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:18.388 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:18.385+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:18.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:263: done (5 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:265: done (4 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:266: done (3 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:268: done (2 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:269: done (1 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:270: done (0 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:271: rmattr oid 8 current snap is 27 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:271: done (0 left) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:272: read oid 10 snap -1 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:272: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-17T16:02:18.423 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 14 current snap is 27 2024-09-17T16:02:18.424 INFO:tasks.rados.rados.0.smithi181.stdout:273: seq_num 112 ranges {1884160=720896,2605056=802816,3407872=360448} 2024-09-17T16:02:18.436 INFO:tasks.rados.rados.0.smithi181.stdout:273: writing smithi18136759-14 from 1884160 to 2605056 tid 1 2024-09-17T16:02:18.440 INFO:tasks.rados.rados.0.smithi181.stdout:273: writing smithi18136759-14 from 2605056 to 3407872 tid 2 2024-09-17T16:02:18.442 INFO:tasks.rados.rados.0.smithi181.stdout:273: writing smithi18136759-14 from 3407872 to 3768320 tid 3 2024-09-17T16:02:18.443 INFO:tasks.rados.rados.0.smithi181.stdout:274: snap_remove snap 7 2024-09-17T16:02:18.468 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:18.468 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.613423589, 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.469 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:02:18.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.61307463, 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:02:18.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:02:18.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.613034499000001, 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.480 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.482 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:02:18.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:18.485 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.600357205000002, 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:02:18.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:02:18.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.600083964, 2024-09-17T16:02:18.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:02:18.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:02:18.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:02:18.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.597218181999999, 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:02:18.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.597166254999998, 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:02:18.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:02:18.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.576625489000001, 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:02:18.508 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:02:18.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:02:18.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:02:18.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.576542314999998, 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.514 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:02:18.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:02:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.562099547999999, 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.519 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:02:18.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.523 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.562059857999998, 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:02:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:02:18.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.526817618000003, 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.530 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:18.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:388 3.16s0 3:6d55c691:::smithi18136759-16:c [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.127646769999998, 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.148899007, 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 388 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338147+0000", 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.129e-06 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338155+0000", 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1389999999999998e-06 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338160+0000", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338288+0000", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128374 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338295+0000", 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7730000000000002e-06 2024-09-17T16:02:18.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.342955+0000", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046598580000000002 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343006+0000", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1329000000000003e-05 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487043+0000", 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14403680199999999 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.16s0 60/53 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.122216061, 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14348618299999999, 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343575+0000", 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6400000000000003e-07 2024-09-17T16:02:18.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343629+0000", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4472000000000003e-05 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486547+0000", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.142917403 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486554+0000", 2024-09-17T16:02:18.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.503e-06 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486609+0000", 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4527000000000002e-05 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487061+0000", 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045181399999999998 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.542 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "replica_scrub(pg: 3.15s1,from:0'0,to:56'8,epoch:60/53,start:3:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.979272122000001, 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45217676400000001, 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:18.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486519+0000", 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486525+0000", 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5949999999999998e-06 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938489+0000", 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45196369600000003 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938505+0000", 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6064000000000001e-05 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938656+0000", 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015132599999999999 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938666+0000", 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6320000000000003e-06 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938695+0000", 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9652e-05 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:447 3.1s0 3:8347559a:::smithi18136759-7:10 [read 97754~16934,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.124923239999999, 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014398077, 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.547 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 447 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340869+0000", 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340875+0000", 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6630000000000004e-06 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340887+0000", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.112e-05 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340896+0000", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344260+0000", 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033644320000000001 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344265+0000", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2479999999999998e-06 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353223+0000", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089583140000000002 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353351+0000", 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128231 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355265+0000", 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019143319999999999 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:448 3.1s0 3:8347559a:::smithi18136759-7:10 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.124867633999999, 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015642311999999998, 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.552 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 448 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:18.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340927+0000", 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.569e-06 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.554 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340936+0000", 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340942+0000", 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344342+0000", 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034007899999999999 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.555 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344344+0000", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.333e-06 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355309+0000", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010965557000000001 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355379+0000", 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9503000000000002e-05 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.556 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356565+0000", 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011865269999999999 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:449 3.1s0 3:8347559a:::smithi18136759-7:10 [read 41111~73577,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 14.124838690000001, 2024-09-17T16:02:18.557 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015941349, 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 449 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4400000000000001e-07 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340955+0000", 2024-09-17T16:02:18.559 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0299999999999998e-06 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340962+0000", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.17e-06 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340968+0000", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344372+0000", 2024-09-17T16:02:18.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034039690000000002 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344373+0000", 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1400000000000001e-06 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355572+0000", 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011199565 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355632+0000", 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9210999999999999e-05 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356893+0000", 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012616330000000001 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.562 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:511 3.6s0 3:60a7d8b5:::smithi18136759-22:head [copy-from ver 12] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.091372129, 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016414522000000001, 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 511 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374421+0000", 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2189999999999998e-06 2024-09-17T16:02:18.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374429+0000", 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4929999999999998e-06 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374449+0000", 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0360000000000002e-05 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374459+0000", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0559999999999994e-06 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374533+0000", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4171999999999994e-05 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374651+0000", 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118495 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.383140+0000", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084888240000000007 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.384289+0000", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011493639999999999 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390716+0000", 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064265609999999999 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390833+0000", 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116978 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/53 ECSubWrite(tid=138, reqid=osd.14.0:138, at_version=74'39, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0618548280000004, 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.11895731, 2024-09-17T16:02:18.568 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403952+0000", 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5957e-05 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:18.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522019+0000", 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.118066429 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522040+0000", 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1769000000000001e-05 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522199+0000", 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015867999999999999 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.571 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522215+0000", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5971999999999999e-05 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522867+0000", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065141500000000002 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522893+0000", 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6505000000000001e-05 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.572 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.573 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.573 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:18.573 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:18.573 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:18.574 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:18.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:18.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:18.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:18.577 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:18.572+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:18.675 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:18.673+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:18.690 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:18.690 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:18.690 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:18.690 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:18.699 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:18.776 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:18.774+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:18.804 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:18.804 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:18.804 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:18.804 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:18.804 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:18.810 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:18.877 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:18.875+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:18.914 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:18.914 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.004563023000003, 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.915 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:02:18.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.003093112000002, 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.919 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:18.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:02:18.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.996740396999996, 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:02:18.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:02:18.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:02:18.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.994757810000003, 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.930 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:02:18.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:18.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.936 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.994331942000002, 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:02:18.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:02:18.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.994303772000002, 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.943 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:02:18.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:18.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:18.949 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.981502919, 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:02:18.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.981183291999997, 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.954 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:02:18.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.981091478000003, 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:02:18.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=30, last_complete=46'18, committed=1, applied=1))", 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.991698176, 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24942447400000001, 2024-09-17T16:02:18.963 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920091+0000", 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6620000000000001e-06 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168910+0000", 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24881899499999999 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168924+0000", 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3968000000000001e-05 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168997+0000", 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3919999999999997e-05 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.169512+0000", 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051455700000000001 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.990486328999999, 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080140109999999997, 2024-09-17T16:02:18.967 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:18.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921300+0000", 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921319+0000", 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8519e-05 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.969 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2544999999999998e-05 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921346+0000", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9420000000000003e-06 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921389+0000", 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2620000000000002e-05 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921394+0000", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3519999999999997e-06 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929283+0000", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007889007 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929313+0000", 2024-09-17T16:02:18.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0460999999999999e-05 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.986781235999999, 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010009911, 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.972 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925020+0000", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925042+0000", 2024-09-17T16:02:18.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2407000000000001e-05 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925050+0000", 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3089999999999998e-06 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925097+0000", 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7763999999999999e-05 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925103+0000", 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0759999999999999e-06 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098665709999999993 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935014+0000", 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4459000000000002e-05 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.976 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 33.982011116000002, 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078830700000000007, 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:18.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929775+0000", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929783+0000", 2024-09-17T16:02:18.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3150000000000003e-06 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929802+0000", 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9959999999999999e-05 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929810+0000", 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3069999999999997e-06 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929880+0000", 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0123000000000006e-05 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929890+0000", 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937625+0000", 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077345599999999997 2024-09-17T16:02:18.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937658+0000", 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2662e-05 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s5 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.572787556000002, 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092966230000000004, 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.981 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338999+0000", 2024-09-17T16:02:18.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1199999999999999e-07 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339027+0000", 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8099999999999999e-05 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339113+0000", 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5958000000000006e-05 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339121+0000", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5949999999999999e-06 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339202+0000", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1150999999999996e-05 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339213+0000", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0607e-05 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348258+0000", 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090445890000000004 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348295+0000", 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7311000000000002e-05 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.56964717, 2024-09-17T16:02:18.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092835030000000002, 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342160+0000", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0803999999999999e-05 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342475+0000", 2024-09-17T16:02:18.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031531799999999998 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342484+0000", 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1609999999999992e-06 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342552+0000", 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7761000000000004e-05 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342562+0000", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0316000000000001e-05 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351383+0000", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088202019999999992 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351422+0000", 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9530000000000003e-05 2024-09-17T16:02:18.989 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.13s4 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.565517274000001, 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081316900000000004, 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346269+0000", 2024-09-17T16:02:18.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346318+0000", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8843e-05 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346402+0000", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4646000000000002e-05 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346411+0000", 2024-09-17T16:02:18.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3890000000000005e-06 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346518+0000", 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107693 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346525+0000", 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5200000000000003e-06 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354365+0000", 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078396709999999994 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354400+0000", 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5627999999999998e-05 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 64/32 ECSubWrite(tid=68, reqid=osd.0.0:68, at_version=64'32, trim_to=0'0, roll_forward_to=62'30))", 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:18.994 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.565416503000002, 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081747250000000007, 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:18.995 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346370+0000", 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6e-07 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346392+0000", 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2419e-05 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:18.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346501+0000", 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010902 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346509+0000", 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0609999999999996e-06 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346569+0000", 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0405999999999997e-05 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346579+0000", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4700000000000008e-06 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354505+0000", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079256489999999999 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354544+0000", 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9440000000000002e-05 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:18.998 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 68/32 ECSubWriteReply(tid=68, last_complete=68'49, committed=1, applied=1))", 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.518056023, 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44479439100000001, 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:18.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7499999999999999e-07 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393734+0000", 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9659999999999998e-06 2024-09-17T16:02:19.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837925+0000", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44419083199999998 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837936+0000", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0349e-05 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838002+0000", 2024-09-17T16:02:19.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6196000000000001e-05 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838524+0000", 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052227299999999996 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:634 3.es0 3:7b038293:::smithi18136759-50:head [stat,delete] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.483085612, 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010543408000000001, 2024-09-17T16:02:19.002 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 634 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428702+0000", 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9709999999999998e-06 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428710+0000", 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2339999999999994e-06 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428728+0000", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7623000000000001e-05 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428736+0000", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2309999999999996e-06 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428803+0000", 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7249000000000006e-05 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428809+0000", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7819999999999999e-06 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.431048+0000", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022392660000000002 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.431135+0000", 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6567e-05 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.432031+0000", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00089615400000000003 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.434430+0000", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023985130000000001 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439144+0000", 2024-09-17T16:02:19.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047146560000000002 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439244+0000", 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9161999999999997e-05 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:639 3.es0 3:7408d595:::smithi18136759-48:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.472131936, 2024-09-17T16:02:19.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080763160000000001, 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 639 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:19.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:19.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439656+0000", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9199999999999998e-06 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439660+0000", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6179999999999999e-06 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439663+0000", 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8210000000000002e-06 2024-09-17T16:02:19.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.442607+0000", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029446619999999998 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.442716+0000", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109127 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.443266+0000", 2024-09-17T16:02:19.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00054992799999999998 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.445211+0000", 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00194429 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.447666+0000", 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002455704 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.014 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.447730+0000", 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3910000000000003e-05 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.015 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.017 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:19.017 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:19.014+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:19.117 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:19.116+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:19.122 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.122 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:19.122 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:19.122 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.130 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:19.219 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:19.217+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:19.234 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.234 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:19.234 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:19.234 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:19.234 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:19.327 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:19.325+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:19.387 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=150, reqid=client.4560.0:177, at_version=38'13, trim_to=0'0, roll_forward_to=38'9))", 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 42.500388090000001, 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038886110000000002, 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884685+0000", 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:19.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884707+0000", 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2036999999999999e-05 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884745+0000", 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7843999999999999e-05 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884756+0000", 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1240000000000001e-05 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884826+0000", 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9873e-05 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.884836+0000", 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0053e-05 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888486+0000", 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036497270000000002 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.888573+0000", 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7068000000000001e-05 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 42.498778295000001, 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:19.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:02:19.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=152, reqid=client.4560.0:183, at_version=38'15, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:19.397 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 42.497967052, 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044954000000000001, 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887106+0000", 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3200000000000004e-07 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887131+0000", 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4892000000000002e-05 2024-09-17T16:02:19.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887167+0000", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5617999999999997e-05 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887176+0000", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6379999999999993e-06 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887237+0000", 2024-09-17T16:02:19.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1760000000000005e-05 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.887246+0000", 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9290000000000004e-06 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891572+0000", 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004325626 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891601+0000", 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9105e-05 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.460973418999998, 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054480279999999997, 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.402 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924122+0000", 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2844999999999999e-05 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924159+0000", 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6534e-05 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924166+0000", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.742e-06 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924210+0000", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4641999999999997e-05 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924220+0000", 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7580000000000004e-06 2024-09-17T16:02:19.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929512+0000", 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005292265 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929547+0000", 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5046999999999998e-05 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=51, reqid=osd.12.0:51, at_version=46'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.460866281000001, 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053997719999999997, 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924207+0000", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924223+0000", 2024-09-17T16:02:19.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6835000000000001e-05 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924252+0000", 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8285000000000001e-05 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924259+0000", 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924320+0000", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1709000000000002e-05 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924329+0000", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8799999999999997e-06 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929582+0000", 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052531169999999999 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929606+0000", 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3913e-05 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.460044041000003, 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:02:19.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.bs4 46/32 ECSubWrite(tid=70, reqid=osd.13.0:67, at_version=0'0, trim_to=0'0, roll_forward_to=46'14))", 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.459716655000001, 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0042172039999999996, 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.416 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925356+0000", 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925357+0000", 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9800000000000003e-07 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925366+0000", 2024-09-17T16:02:19.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1900000000000001e-06 2024-09-17T16:02:19.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925396+0000", 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9964e-05 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925404+0000", 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6429999999999995e-06 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925470+0000", 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6817999999999996e-05 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925480+0000", 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8509999999999994e-06 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929543+0000", 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040623580000000003 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929573+0000", 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0682000000000002e-05 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.457685079000001, 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:02:19.422 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:02:19.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.426 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.455284437000003, 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052541769999999996, 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929789+0000", 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:02:19.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929800+0000", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1341e-05 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929829+0000", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8642000000000001e-05 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929837+0000", 2024-09-17T16:02:19.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929901+0000", 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4067999999999997e-05 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929907+0000", 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.609e-06 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934990+0000", 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050827579999999997 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935042+0000", 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2676999999999997e-05 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=39, reqid=osd.0.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:19.431 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 34.452524283000002, 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050006720000000003, 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932548+0000", 2024-09-17T16:02:19.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932549+0000", 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6100000000000001e-07 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932561+0000", 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1674e-05 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932581+0000", 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0024999999999999e-05 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932590+0000", 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7780000000000002e-06 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932642+0000", 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1753999999999999e-05 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932651+0000", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2310000000000002e-06 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937518+0000", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0048669969999999996 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.937549+0000", 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0651999999999997e-05 2024-09-17T16:02:19.435 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 29.033816201, 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043350350000000001, 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351256+0000", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351257+0000", 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0100000000000001e-07 2024-09-17T16:02:19.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351295+0000", 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8333e-05 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351337+0000", 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1491e-05 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351345+0000", 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5189999999999992e-06 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351424+0000", 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.351434+0000", 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9760000000000007e-06 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355546+0000", 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041123100000000001 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.355591+0000", 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5212e-05 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.440 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.7s1 52/32 ECSubWrite(tid=85, reqid=osd.13.0:78, at_version=0'0, trim_to=0'0, roll_forward_to=52'18))", 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 29.014731399999999, 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037161669999999998, 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370341+0000", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.17e-07 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370349+0000", 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4499999999999998e-06 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370413+0000", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4089000000000003e-05 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370419+0000", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2740000000000004e-06 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370550+0000", 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000131163 2024-09-17T16:02:19.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.370561+0000", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0550999999999999e-05 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374029+0000", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034684590000000001 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.374057+0000", 2024-09-17T16:02:19.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8064000000000001e-05 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.039335356999999, 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060163380000000004, 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.445 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8900000000000001e-07 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345763+0000", 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5772e-05 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345830+0000", 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6855999999999998e-05 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345839+0000", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345913+0000", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3454999999999994e-05 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345924+0000", 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:19.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351719+0000", 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057956309999999999 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351754+0000", 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4439e-05 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.039306527000001, 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087141639999999999, 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345787+0000", 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0109e-05 2024-09-17T16:02:19.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346312+0000", 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052588999999999999 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346324+0000", 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1445e-05 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346394+0000", 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9604000000000006e-05 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346403+0000", 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.611e-06 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354448+0000", 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080448199999999994 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354480+0000", 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2329000000000002e-05 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.038874189000001, 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082305039999999996, 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.454 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346199+0000", 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7300000000000002e-07 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346223+0000", 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4442000000000001e-05 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346360+0000", 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136792 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346374+0000", 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3669000000000001e-05 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346429+0000", 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4898999999999997e-05 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346437+0000", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354393+0000", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079559079999999994 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354429+0000", 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6168999999999999e-05 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.458 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.10s1 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.038835188, 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083156649999999999, 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346238+0000", 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346256+0000", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8805999999999999e-05 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346408+0000", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000151519 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346417+0000", 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6619999999999994e-06 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346513+0000", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6127999999999998e-05 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346523+0000", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9750000000000002e-06 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354521+0000", 2024-09-17T16:02:19.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079977780000000005 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354553+0000", 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2614000000000003e-05 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 64/32 ECSubWrite(tid=130, reqid=osd.12.0:127, at_version=0'0, trim_to=0'0, roll_forward_to=64'43))", 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:19.463 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.037855914000001, 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0045663939999999997, 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347217+0000", 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5499999999999998e-07 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347225+0000", 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7279999999999995e-06 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347280+0000", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5435e-05 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347288+0000", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4510000000000003e-06 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347350+0000", 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2533000000000006e-05 2024-09-17T16:02:19.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.347359+0000", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4349999999999997e-06 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351754+0000", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0043954759999999997 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351783+0000", 2024-09-17T16:02:19.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8881e-05 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 69/53 ECSubWrite(tid=216, reqid=client.4560.0:480, at_version=69'34, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.038338227000001, 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0049892540000000003, 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.468 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346734+0000", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.346735+0000", 2024-09-17T16:02:19.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5099999999999999e-07 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347399+0000", 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00066419799999999996 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347439+0000", 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9925000000000001e-05 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347447+0000", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7159999999999992e-06 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347504+0000", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6396e-05 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.347514+0000", 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0526999999999999e-05 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351687+0000", 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0041724650000000002 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351724+0000", 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6875999999999997e-05 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 69/53 ECSubWrite(tid=217, reqid=client.4560.0:481, at_version=69'35, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 12.034403633, 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039025790000000002, 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350669+0000", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350670+0000", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0100000000000003e-07 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.350992+0000", 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321758 2024-09-17T16:02:19.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351026+0000", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4221000000000003e-05 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351036+0000", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0044000000000001e-05 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351104+0000", 2024-09-17T16:02:19.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8035999999999997e-05 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.351114+0000", 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9059999999999997e-06 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.354535+0000", 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0034212610000000001 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.354572+0000", 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6452000000000003e-05 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 79/32 ECSubWrite(tid=97, reqid=client.4560.0:643, at_version=79'44, trim_to=0'0, roll_forward_to=77'43))", 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:17.440788+0000", 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 1.944285126, 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0037210120000000001, 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.477 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440788+0000", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440788+0000", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440788+0000", 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7700000000000001e-07 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440806+0000", 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8289999999999999e-05 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440844+0000", 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8157000000000001e-05 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440862+0000", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8031000000000001e-05 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440932+0000", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0292999999999999e-05 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.440944+0000", 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1318000000000001e-05 2024-09-17T16:02:19.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.444467+0000", 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00352301 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.444509+0000", 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1736000000000002e-05 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.482 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.482 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.484 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:02:19.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:274: done (2 left) 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 42 current snap is 27 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:275: seq_num 113 ranges {1867776=737280,2605056=557056,3162112=606208} 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18136759-42 from 1867776 to 2605056 tid 1 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18136759-42 from 2605056 to 3162112 tid 2 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:275: writing smithi18136759-42 from 3162112 to 3768320 tid 3 2024-09-17T16:02:19.485 INFO:tasks.rados.rados.0.smithi181.stdout:272: expect (ObjNum 109 snap 26 seq_num 109) 2024-09-17T16:02:19.486 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:19.486 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:19.481+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:19.499 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:19.544 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:19.584 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:19.582+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:273: finishing write tid 1 to smithi18136759-14 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:273: finishing write tid 2 to smithi18136759-14 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:272: done (2 left) 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:276: read oid 10 snap 4 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:276: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-17T16:02:19.589 INFO:tasks.rados.rados.0.smithi181.stdout:277: snap_create 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: finishing write tid 3 to smithi18136759-14 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: finishing write tid 4 to smithi18136759-14 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: finishing write tid 5 to smithi18136759-14 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: oid 14 updating version 0 to 156 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: oid 14 updating version 156 to 159 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: oid 14 version 159 is already newer than 158 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: oid 14 version 159 is already newer than 157 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 14 v 159 (ObjNum 112 snap 27 seq_num 112) dirty exists 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:273: left oid 14 (ObjNum 112 snap 27 seq_num 112) 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 1 to smithi18136759-42 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 2 to smithi18136759-42 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 3 to smithi18136759-42 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 4 to smithi18136759-42 2024-09-17T16:02:19.590 INFO:tasks.rados.rados.0.smithi181.stdout:275: finishing write tid 5 to smithi18136759-42 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 42 updating version 0 to 41 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 42 updating version 41 to 44 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 42 version 44 is already newer than 42 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:275: oid 42 version 44 is already newer than 43 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 42 v 44 (ObjNum 113 snap 27 seq_num 113) dirty exists 2024-09-17T16:02:19.591 INFO:tasks.rados.rados.0.smithi181.stdout:275: left oid 42 (ObjNum 113 snap 27 seq_num 113) 2024-09-17T16:02:19.593 INFO:tasks.rados.rados.0.smithi181.stdout:276: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-17T16:02:19.593 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 11705 2024-09-17T16:02:19.594 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 12554 2024-09-17T16:02:19.596 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 123873 2024-09-17T16:02:19.629 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.629 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:19.629 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:19.629 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:19.722 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:19.720+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:19.780 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.781 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:19.781 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:19.781 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:19.781 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:19.785 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:19.783+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:19.789 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:19.886 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:19.884+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:19.935 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.077402607000003, 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.936 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:02:19.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:02:19.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.940 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.076501733000001, 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:02:19.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:19.945 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.072411905999999, 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:02:19.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.070004945000001, 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.950 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:02:19.952 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.953 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.954 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.068179567000001, 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.955 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.956 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.957 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.958 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.959 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.068031492000003, 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.960 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:02:19.961 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.962 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:02:19.963 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.067381607999998, 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:02:19.964 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:19.965 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.966 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.967 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:02:19.968 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.064804244999998, 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.969 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:19.970 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.971 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:02:19.972 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.973 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.064313073000001, 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:19.974 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.975 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:19.976 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:02:19.977 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 52/32 ECSubWrite(tid=89, reqid=client.4560.0:340, at_version=52'11, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:19.978 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 29.559391663, 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15958299200000001, 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.979 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372994+0000", 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373011+0000", 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6280999999999999e-05 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.980 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531935+0000", 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15892432300000001 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531949+0000", 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.981 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532026+0000", 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6526000000000006e-05 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532033+0000", 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532538+0000", 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050494099999999996 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.982 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532576+0000", 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8059999999999998e-05 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 17.593867193000001, 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058173629999999999, 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.983 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.984 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5199999999999997e-07 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338580+0000", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1075999999999996e-05 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338633+0000", 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3263999999999997e-05 2024-09-17T16:02:19.985 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338646+0000", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3326000000000001e-05 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339218+0000", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057138000000000002 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339229+0000", 2024-09-17T16:02:19.986 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1487e-05 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344292+0000", 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050634460000000001 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344335+0000", 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2831999999999999e-05 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.987 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:602 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~606208] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.5007675039999997, 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085762449999999997, 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:19.988 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 602 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.989 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431620+0000", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5469999999999999e-06 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432327+0000", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000707079 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432340+0000", 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3021e-05 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.990 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432358+0000", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8019000000000001e-05 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432480+0000", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121995 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432491+0000", 2024-09-17T16:02:19.991 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0457e-05 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434624+0000", 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021328250000000001 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434764+0000", 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014050999999999999 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.992 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.436049+0000", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012844969999999999 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439929+0000", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038802730000000001 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440073+0000", 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014401099999999999 2024-09-17T16:02:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440194+0000", 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121011 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:603 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~688128] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4999571950000004, 2024-09-17T16:02:19.994 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089592730000000002, 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 603 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:19.995 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432429+0000", 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.996 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432988+0000", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055916400000000002 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432997+0000", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8310000000000007e-06 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433001+0000", 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:02:19.997 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433056+0000", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4994999999999998e-05 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433061+0000", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437100+0000", 2024-09-17T16:02:19.998 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040393169999999997 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437171+0000", 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0548000000000003e-05 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437907+0000", 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073646200000000001 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:19.999 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439930+0000", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002022565 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441283+0000", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013528189999999999 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441387+0000", 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010435 2024-09-17T16:02:20.000 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:604 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~475136] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4993698740000001, 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086384329999999992, 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.001 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 604 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.002 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433016+0000", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6899999999999997e-07 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433437+0000", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042084799999999998 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433445+0000", 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.354e-06 2024-09-17T16:02:20.003 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433450+0000", 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1460000000000004e-06 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433526+0000", 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5370000000000005e-05 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:20.004 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.22e-06 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438550+0000", 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050189079999999999 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438619+0000", 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9138000000000001e-05 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.005 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439350+0000", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073165800000000005 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441219+0000", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001868747 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441548+0000", 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032912700000000002 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.006 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441654+0000", 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010534799999999999 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:605 3.11s0 3:89c7af4b:::smithi18136759-34:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4989240930000003, 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086379049999999995, 2024-09-17T16:02:20.007 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 605 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.008 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.009 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433484+0000", 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2540000000000001e-05 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433488+0000", 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1790000000000001e-06 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433491+0000", 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.915e-06 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.010 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433560+0000", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9307000000000006e-05 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433565+0000", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8049999999999997e-06 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440435+0000", 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068700439999999996 2024-09-17T16:02:20.011 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440548+0000", 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112904 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440860+0000", 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031159299999999998 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.012 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441710+0000", 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085002099999999996 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441963+0000", 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025321099999999999 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442099+0000", 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013592399999999999 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.013 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:606 3.11s0 3:89c7af4b:::smithi18136759-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.4988660170000001, 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010718325000000001, 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 606 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.014 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.015 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.016 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:20.016 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6800000000000001e-07 2024-09-17T16:02:20.016 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.016 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.016 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433522+0000", 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.052e-06 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9580000000000002e-06 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.017 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433536+0000", 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4750000000000004e-06 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433588+0000", 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2129999999999997e-05 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433592+0000", 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9090000000000002e-06 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.018 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441080+0000", 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007488476 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441139+0000", 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8468000000000003e-05 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.019 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442343+0000", 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012042520000000001 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442403+0000", 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9846999999999998e-05 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.444237+0000", 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00183439 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.020 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=148, reqid=osd.14.0:148, at_version=81'37, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.49424450599999997, 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065669389999999999, 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.021 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.022 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9900000000000002e-07 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438158+0000", 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7248000000000002e-05 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.023 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438229+0000", 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1317999999999997e-05 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438248+0000", 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.827e-05 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438320+0000", 2024-09-17T16:02:20.024 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2342999999999995e-05 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438331+0000", 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444669+0000", 2024-09-17T16:02:20.025 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063379129999999997 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444708+0000", 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8578999999999998e-05 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.026 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=150, reqid=osd.14.0:150, at_version=81'39, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.49302641200000002, 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057892539999999998, 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.027 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:20.028 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6700000000000003e-07 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439380+0000", 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0604000000000001e-05 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439418+0000", 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7707000000000003e-05 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.029 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439426+0000", 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3289999999999994e-06 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439492+0000", 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5801000000000002e-05 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.030 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.031 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439505+0000", 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2699e-05 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445102+0000", 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055975479999999999 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.032 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445148+0000", 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5899000000000002e-05 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.48678117799999998, 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082580629999999995, 2024-09-17T16:02:20.033 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.034 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445605+0000", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029e-06 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445725+0000", 2024-09-17T16:02:20.035 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120361 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445763+0000", 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7991000000000003e-05 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445773+0000", 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0039e-05 2024-09-17T16:02:20.036 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445830+0000", 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6909e-05 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445841+0000", 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0327e-05 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.037 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453823+0000", 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079825799999999995 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453862+0000", 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8826999999999999e-05 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.038 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.48374445799999999, 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055985469999999997, 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.039 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.040 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448735+0000", 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3511000000000006e-05 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448773+0000", 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8169e-05 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448782+0000", 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7509999999999997e-06 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.041 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448846+0000", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4108000000000004e-05 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448877+0000", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1097999999999999e-05 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454215+0000", 2024-09-17T16:02:20.042 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053383129999999999 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454239+0000", 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3830000000000001e-05 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.043 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:20.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:20.046 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:20.043+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:20.145 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:20.143+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:20.190 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:20.190 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:20.190 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:20.190 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:20.199 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:20.283 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:20.281+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:20.341 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:20.341 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:20.341 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:20.341 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:20.342 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:20.347 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:20.345+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:20.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:20.429 INFO:tasks.rados.rados.0.smithi181.stdout:273: done (3 left) 2024-09-17T16:02:20.429 INFO:tasks.rados.rados.0.smithi181.stdout:275: done (2 left) 2024-09-17T16:02:20.429 INFO:tasks.rados.rados.0.smithi181.stdout:276: done (1 left) 2024-09-17T16:02:20.430 INFO:tasks.rados.rados.0.smithi181.stdout:277: done (0 left) 2024-09-17T16:02:20.430 INFO:tasks.rados.rados.0.smithi181.stdout:278: snap_remove snap 11 2024-09-17T16:02:20.448 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:20.446+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:20.494 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:20.494 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:20.494 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.639251414999997, 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:02:20.495 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.496 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.497 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.498 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.499 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.639222902, 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.500 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:20.501 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.502 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.503 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.504 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:20.505 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.639083857000003, 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.506 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:02:20.507 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.508 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:02:20.509 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.510 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:02:20.511 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:02:20.511 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.639056183000001, 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:02:20.512 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.513 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.514 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.515 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:02:20.516 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.517 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.574619605000002, 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.518 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:02:20.519 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.520 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:02:20.521 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.522 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.523 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.574594400999999, 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.524 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:20.525 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.526 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.527 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.528 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.529 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.571228407, 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.530 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.531 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.532 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:02:20.533 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.544253757999996, 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:02:20.534 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.535 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.536 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:02:20.537 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:02:20.538 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.539 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.544235886999999, 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.540 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:20.541 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.542 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.543 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.544 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 46.540942119999997, 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.545 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:02:20.546 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:02:20.547 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.548 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.540826365000001, 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.549 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.550 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.551 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.552 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 35.560972984999999, 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:20.553 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.554 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.555 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 55/53 ECSubWriteReply(tid=63, last_complete=55'29, committed=1, applied=1))", 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.115057746000002, 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.850069713, 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.556 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.557 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376978+0000", 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376981+0000", 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226831+0000", 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84985070900000004 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.558 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226846+0000", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4192999999999999e-05 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226976+0000", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013074699999999999 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.227047+0000", 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0347999999999998e-05 2024-09-17T16:02:20.559 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 27.098858249999999, 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023393622999999999, 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.560 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:20.561 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.82e-07 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393970+0000", 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079326400000000001 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414706+0000", 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020736065000000001 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.562 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414712+0000", 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414788+0000", 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6206999999999997e-05 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414798+0000", 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.563 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416538+0000", 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00174007 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416570+0000", 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2415000000000001e-05 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.564 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpReadReply(3.0s0 68/53 ECSubReadReply(tid=92, attrs_read=0))", 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.633710975, 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.495310574, 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.565 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.566 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.859357+0000", 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010325270000000001 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352866+0000", 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49350909799999998 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352872+0000", 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5819999999999998e-06 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.567 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352920+0000", 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7234999999999998e-05 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.353634+0000", 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071472599999999995 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 69/53 ECSubWriteReply(tid=93, last_complete=69'46, committed=1, applied=1))", 2024-09-17T16:02:20.568 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.139196245999999, 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02328291, 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.569 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352839+0000", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4e-07 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352844+0000", 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6619999999999996e-06 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.570 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375190+0000", 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.022345941000000001 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375196+0000", 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.571 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375242+0000", 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6655e-05 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376121+0000", 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00087903899999999997 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:497 3.9s0 3:905994b3:::smithi18136759-13:head [append 0~0] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:20.572 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.118514569, 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.035534871000000003, 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 497 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.573 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373521+0000", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8200000000000001e-07 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.574 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373523+0000", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.272e-06 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373529+0000", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2750000000000001e-06 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373534+0000", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0359999999999997e-06 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.575 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375754+0000", 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002219834 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375866+0000", 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011218900000000001 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376638+0000", 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077170400000000001 2024-09-17T16:02:20.576 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380515+0000", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038770889999999998 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408971+0000", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02845599 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409055+0000", 2024-09-17T16:02:20.577 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4099999999999998e-05 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 69/53 ECSubWriteReply(tid=97, last_complete=69'23, committed=1, applied=1))", 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.111770629, 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029135165000000001, 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.578 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.579 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380265+0000", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408819+0000", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028553388999999998 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408826+0000", 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0140000000000001e-06 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.580 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408882+0000", 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5704000000000001e-05 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409399+0000", 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051776899999999995 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 69/53 ECSubWriteReply(tid=104, last_complete=69'50, committed=1, applied=1))", 2024-09-17T16:02:20.581 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 13.072300163, 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35452754399999997, 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.582 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419738+0000", 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.65e-06 2024-09-17T16:02:20.583 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774110+0000", 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35437182699999997 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774122+0000", 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2642e-05 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.584 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774218+0000", 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5758e-05 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774262+0000", 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4205000000000001e-05 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 74/53 ECSubWriteReply(tid=138, last_complete=74'39, committed=1, applied=1))", 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 8.0874380319999997, 2024-09-17T16:02:20.585 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36342225099999997, 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.586 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8000000000000002e-07 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404599+0000", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7549999999999999e-06 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767461+0000", 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36286232000000002 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.587 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767477+0000", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5934e-05 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767592+0000", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011500199999999999 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.768019+0000", 2024-09-17T16:02:20.588 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042695999999999999 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:20.589 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:20.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:20.591 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:20.589+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:20.591 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:20.691 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:20.689+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:20.700 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:20.700 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:20.700 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:20.700 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:20.709 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:20.737 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:20.738 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:20.761 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:20.761 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:20.782 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:20.783 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:20.792 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:20.790+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:20.794 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:20.794 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:20.794 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-17T16:02:20.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:02:20.804 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:20.804 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:20.812 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:20.812 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:20.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:20.825 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:20.825 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:20.847 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:20.847 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:20.892 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:20.890+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:20.923 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:20.924 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:20.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:20.966 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:20.966 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:20.987 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:20.987 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:20.993 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:20.991+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:21.010 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:21.010 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:21.031 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:21.031 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:21.036 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:21.036 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:21.036 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:21.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:21.052 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:21.052 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:21.072 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:21.073 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:21.094 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:21.092+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:21.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:21.095 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:21.105 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:21.105 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:02:21.116 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:21.116 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:21.117 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-17T16:02:20.950248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":91619891,"num_objects":140,"num_object_clones":86,"num_object_copies":960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":9,"num_read":286,"num_read_kb":67058,"num_write":518,"num_write_kb":65906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1213,"ondisk_log_size":1213,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":635036,"kb_used_data":165100,"kb_used_omap":24,"kb_used_meta":469543,"kb_avail":1509314404,"statfs":{"total":1546188226560,"available":1545537949696,"internally_reserved":0,"allocated":169062400,"data_stored":243067550,"data_compressed":2780222,"data_compressed_allocated":3780608,"data_compressed_original":8437760,"omap_allocated":25519,"internal_metadata":480812113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7684096,"num_objects":4,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":5375,"num_write":67,"num_write_kb":7952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001186"},"pg_stats":[{"pgid":"3.19","version":"81'44","reported_seq":313,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.458970+0000","last_change":"2024-09-17T16:02:19.449529+0000","last_active":"2024-09-17T16:02:19.458970+0000","last_peered":"2024-09-17T16:02:19.458970+0000","last_clean":"2024-09-17T16:02:19.458970+0000","last_became_active":"2024-09-17T16:01:52.357491+0000","last_became_peered":"2024-09-17T16:01:52.357491+0000","last_unstale":"2024-09-17T16:02:19.458970+0000","last_undegraded":"2024-09-17T16:02:19.458970+0000","last_fullsized":"2024-09-17T16:02:19.458970+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:01:59.252035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.252035+0000","objects_scrubbed":4,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:09.060144+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.019818234000000001,"stat_sum":{"num_bytes":4292608,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.18","version":"81'62","reported_seq":325,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.442214+0000","last_change":"2024-09-17T16:02:19.446080+0000","last_active":"2024-09-17T16:02:19.442214+0000","last_peered":"2024-09-17T16:02:19.442214+0000","last_clean":"2024-09-17T16:02:19.442214+0000","last_became_active":"2024-09-17T16:01:53.177881+0000","last_became_peered":"2024-09-17T16:01:53.177881+0000","last_unstale":"2024-09-17T16:02:19.442214+0000","last_undegraded":"2024-09-17T16:02:19.442214+0000","last_fullsized":"2024-09-17T16:02:19.442214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'31","last_scrub_stamp":"2024-09-17T16:02:00.137409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.137409+0000","objects_scrubbed":2,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:06.964749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.016540216999999999,"stat_sum":{"num_bytes":5603328,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":13,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.17","version":"73'38","reported_seq":267,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.430672+0000","last_change":"2024-09-17T16:02:04.352378+0000","last_active":"2024-09-17T16:02:19.430672+0000","last_peered":"2024-09-17T16:02:19.430672+0000","last_clean":"2024-09-17T16:02:02.351935+0000","last_became_active":"2024-09-17T16:02:04.352378+0000","last_became_peered":"2024-09-17T16:02:04.352378+0000","last_unstale":"2024-09-17T16:02:19.430672+0000","last_undegraded":"2024-09-17T16:02:04.343159+0000","last_fullsized":"2024-09-17T16:02:04.342807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'27","last_scrub_stamp":"2024-09-17T16:02:02.077678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.077678+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:22.065731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.027309317,"stat_sum":{"num_bytes":3883008,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":818,"num_write":6,"num_write_kb":1680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":5}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.16","version":"79'30","reported_seq":272,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.430644+0000","last_change":"2024-09-17T16:02:04.351696+0000","last_active":"2024-09-17T16:02:19.430644+0000","last_peered":"2024-09-17T16:02:19.430644+0000","last_clean":"2024-09-17T16:02:02.354687+0000","last_became_active":"2024-09-17T16:02:04.351696+0000","last_became_peered":"2024-09-17T16:02:04.351696+0000","last_unstale":"2024-09-17T16:02:19.430644+0000","last_undegraded":"2024-09-17T16:02:04.341016+0000","last_fullsized":"2024-09-17T16:02:04.340608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'16","last_scrub_stamp":"2024-09-17T16:02:00.050173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.050173+0000","objects_scrubbed":4,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:09.087711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01423026,"stat_sum":{"num_bytes":6619136,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7052,"num_write":8,"num_write_kb":3232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.15","version":"81'22","reported_seq":291,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.446494+0000","last_change":"2024-09-17T16:02:19.448089+0000","last_active":"2024-09-17T16:02:19.446494+0000","last_peered":"2024-09-17T16:02:19.446494+0000","last_clean":"2024-09-17T16:02:19.446494+0000","last_became_active":"2024-09-17T16:01:53.142297+0000","last_became_peered":"2024-09-17T16:01:53.142297+0000","last_unstale":"2024-09-17T16:02:19.446494+0000","last_undegraded":"2024-09-17T16:02:19.446494+0000","last_fullsized":"2024-09-17T16:02:19.446494+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'8","last_scrub_stamp":"2024-09-17T16:01:58.940426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.940426+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.978796+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0.014126290999999999,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.14","version":"79'28","reported_seq":295,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-17T16:02:18.605884+0000","last_change":"2024-09-17T16:02:17.435122+0000","last_active":"2024-09-17T16:02:18.605884+0000","last_peered":"2024-09-17T16:02:18.605884+0000","last_clean":"2024-09-17T16:02:18.605884+0000","last_became_active":"2024-09-17T16:01:52.360122+0000","last_became_peered":"2024-09-17T16:01:52.360122+0000","last_unstale":"2024-09-17T16:02:18.605884+0000","last_undegraded":"2024-09-17T16:02:18.605884+0000","last_fullsized":"2024-09-17T16:02:18.605884+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:58.496684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.496684+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:25.902232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0044632099999999996,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.430359+0000","last_change":"2024-09-17T16:01:58.347762+0000","last_active":"2024-09-17T16:02:19.430359+0000","last_peered":"2024-09-17T16:02:19.430359+0000","last_clean":"2024-09-17T16:02:19.430359+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:02:19.430359+0000","last_undegraded":"2024-09-17T16:02:19.430359+0000","last_fullsized":"2024-09-17T16:02:19.430359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:30.740032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"75'15","reported_seq":245,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.431553+0000","last_change":"2024-09-17T16:02:04.349120+0000","last_active":"2024-09-17T16:02:19.431553+0000","last_peered":"2024-09-17T16:02:19.431553+0000","last_clean":"2024-09-17T16:02:02.355614+0000","last_became_active":"2024-09-17T16:02:04.349120+0000","last_became_peered":"2024-09-17T16:02:04.349120+0000","last_unstale":"2024-09-17T16:02:19.431553+0000","last_undegraded":"2024-09-17T16:02:04.340835+0000","last_fullsized":"2024-09-17T16:02:04.340587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:02:01.398192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.398192+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:02.781632+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0.018760373,"stat_sum":{"num_bytes":786432,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.431125+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:02:19.431125+0000","last_peered":"2024-09-17T16:02:19.431125+0000","last_clean":"2024-09-17T16:02:19.431125+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:02:19.431125+0000","last_undegraded":"2024-09-17T16:02:19.431125+0000","last_fullsized":"2024-09-17T16:02:19.431125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"72'22","reported_seq":241,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.431696+0000","last_change":"2024-09-17T16:02:04.349605+0000","last_active":"2024-09-17T16:02:19.431696+0000","last_peered":"2024-09-17T16:02:19.431696+0000","last_clean":"2024-09-17T16:02:03.036314+0000","last_became_active":"2024-09-17T16:02:04.349605+0000","last_became_peered":"2024-09-17T16:02:04.349605+0000","last_unstale":"2024-09-17T16:02:19.431696+0000","last_undegraded":"2024-09-17T16:02:04.340916+0000","last_fullsized":"2024-09-17T16:02:04.340610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01416717,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":15,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":126,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.432824+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:02:19.432824+0000","last_peered":"2024-09-17T16:02:19.432824+0000","last_clean":"2024-09-17T16:02:19.432824+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:02:19.432824+0000","last_undegraded":"2024-09-17T16:02:19.432824+0000","last_fullsized":"2024-09-17T16:02:19.432824+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":270,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-17T16:02:18.606029+0000","last_change":"2024-09-17T16:02:17.434998+0000","last_active":"2024-09-17T16:02:18.606029+0000","last_peered":"2024-09-17T16:02:18.606029+0000","last_clean":"2024-09-17T16:02:18.606029+0000","last_became_active":"2024-09-17T16:01:52.351079+0000","last_became_peered":"2024-09-17T16:01:52.351079+0000","last_unstale":"2024-09-17T16:02:18.606029+0000","last_undegraded":"2024-09-17T16:02:18.606029+0000","last_fullsized":"2024-09-17T16:02:18.606029+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:59.453974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.453974+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:13.886669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041087129999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":123,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-17T16:02:18.606082+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:02:18.606082+0000","last_peered":"2024-09-17T16:02:18.606082+0000","last_clean":"2024-09-17T16:02:18.606082+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:02:18.606082+0000","last_undegraded":"2024-09-17T16:02:18.606082+0000","last_fullsized":"2024-09-17T16:02:18.606082+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"81'22","reported_seq":285,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.446626+0000","last_change":"2024-09-17T16:02:19.448545+0000","last_active":"2024-09-17T16:02:19.446626+0000","last_peered":"2024-09-17T16:02:19.446626+0000","last_clean":"2024-09-17T16:02:19.446626+0000","last_became_active":"2024-09-17T16:01:52.345743+0000","last_became_peered":"2024-09-17T16:01:52.345743+0000","last_unstale":"2024-09-17T16:02:19.446626+0000","last_undegraded":"2024-09-17T16:02:19.446626+0000","last_fullsized":"2024-09-17T16:02:19.446626+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'8","last_scrub_stamp":"2024-09-17T16:01:59.457065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.457065+0000","objects_scrubbed":2,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:26.499252+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.014483665999999999,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.3","version":"81'33","reported_seq":298,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.446486+0000","last_change":"2024-09-17T16:02:19.447422+0000","last_active":"2024-09-17T16:02:19.446486+0000","last_peered":"2024-09-17T16:02:19.446486+0000","last_clean":"2024-09-17T16:02:19.446486+0000","last_became_active":"2024-09-17T16:01:52.349931+0000","last_became_peered":"2024-09-17T16:01:52.349931+0000","last_unstale":"2024-09-17T16:02:19.446486+0000","last_undegraded":"2024-09-17T16:02:19.446486+0000","last_fullsized":"2024-09-17T16:02:19.446486+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'18","last_scrub_stamp":"2024-09-17T16:01:59.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.099258+0000","objects_scrubbed":4,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:09.342969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.021101292000000001,"stat_sum":{"num_bytes":1212416,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":62,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":128,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.433496+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:02:19.433496+0000","last_peered":"2024-09-17T16:02:19.433496+0000","last_clean":"2024-09-17T16:02:19.433496+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:02:19.433496+0000","last_undegraded":"2024-09-17T16:02:19.433496+0000","last_fullsized":"2024-09-17T16:02:19.433496+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"81'76","reported_seq":329,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.451265+0000","last_change":"2024-09-17T16:02:19.454387+0000","last_active":"2024-09-17T16:02:19.451265+0000","last_peered":"2024-09-17T16:02:19.451265+0000","last_clean":"2024-09-17T16:02:19.451265+0000","last_became_active":"2024-09-17T16:01:52.345493+0000","last_became_peered":"2024-09-17T16:01:52.345493+0000","last_unstale":"2024-09-17T16:02:19.451265+0000","last_undegraded":"2024-09-17T16:02:19.451265+0000","last_fullsized":"2024-09-17T16:02:19.451265+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'27","last_scrub_stamp":"2024-09-17T16:02:00.289587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.289587+0000","objects_scrubbed":4,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:17.173930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.024580629,"stat_sum":{"num_bytes":2850816,"num_objects":8,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":3048,"num_write":9,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":125,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.426081+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:02:19.426081+0000","last_peered":"2024-09-17T16:02:19.426081+0000","last_clean":"2024-09-17T16:02:19.426081+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:02:19.426081+0000","last_undegraded":"2024-09-17T16:02:19.426081+0000","last_fullsized":"2024-09-17T16:02:19.426081+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"72'19","reported_seq":249,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.430264+0000","last_change":"2024-09-17T16:02:04.352734+0000","last_active":"2024-09-17T16:02:19.430264+0000","last_peered":"2024-09-17T16:02:19.430264+0000","last_clean":"2024-09-17T16:02:02.350917+0000","last_became_active":"2024-09-17T16:02:04.352734+0000","last_became_peered":"2024-09-17T16:02:04.352734+0000","last_unstale":"2024-09-17T16:02:19.430264+0000","last_undegraded":"2024-09-17T16:02:04.341137+0000","last_fullsized":"2024-09-17T16:02:04.340823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.086155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.086155+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:16.840775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014571756,"stat_sum":{"num_bytes":2080768,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":202,"num_write":3,"num_write_kb":800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,2147483647],"acting":[13,4,3,11,10,2,2147483647],"avail_no_missing":["13(0)","2(5)","3(2)","4(1)","10(4)","11(3)"],"object_location_counts":[{"shards":"2(5),3(2),4(1),10(4),11(3),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":126,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.433337+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:02:19.433337+0000","last_peered":"2024-09-17T16:02:19.433337+0000","last_clean":"2024-09-17T16:02:19.433337+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:02:19.433337+0000","last_undegraded":"2024-09-17T16:02:19.433337+0000","last_fullsized":"2024-09-17T16:02:19.433337+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"79'47","reported_seq":278,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.429861+0000","last_change":"2024-09-17T16:02:04.349933+0000","last_active":"2024-09-17T16:02:19.429861+0000","last_peered":"2024-09-17T16:02:19.429861+0000","last_clean":"2024-09-17T16:02:02.354956+0000","last_became_active":"2024-09-17T16:02:04.349933+0000","last_became_peered":"2024-09-17T16:02:04.349933+0000","last_unstale":"2024-09-17T16:02:19.429861+0000","last_undegraded":"2024-09-17T16:02:04.341956+0000","last_fullsized":"2024-09-17T16:02:04.341160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'35","last_scrub_stamp":"2024-09-17T16:02:01.159864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.159864+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:21.751979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023769390000000001,"stat_sum":{"num_bytes":4030464,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":138,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,2147483647],"acting":[11,2,0,4,8,3,2147483647],"avail_no_missing":["11(0)","0(2)","2(1)","3(5)","4(3)","8(4)"],"object_location_counts":[{"shards":"0(2),2(1),3(5),4(3),8(4),11(0)","objects":10}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":127,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.429481+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:02:19.429481+0000","last_peered":"2024-09-17T16:02:19.429481+0000","last_clean":"2024-09-17T16:02:19.429481+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:02:19.429481+0000","last_undegraded":"2024-09-17T16:02:19.429481+0000","last_fullsized":"2024-09-17T16:02:19.429481+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":261,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.430023+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:02:19.430023+0000","last_peered":"2024-09-17T16:02:19.430023+0000","last_clean":"2024-09-17T16:02:19.430023+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:02:19.430023+0000","last_undegraded":"2024-09-17T16:02:19.430023+0000","last_fullsized":"2024-09-17T16:02:19.430023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"81'50","reported_seq":312,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.442652+0000","last_change":"2024-09-17T16:02:19.450238+0000","last_active":"2024-09-17T16:02:19.442652+0000","last_peered":"2024-09-17T16:02:19.442652+0000","last_clean":"2024-09-17T16:02:19.442652+0000","last_became_active":"2024-09-17T16:01:52.347600+0000","last_became_peered":"2024-09-17T16:01:52.347600+0000","last_unstale":"2024-09-17T16:02:19.442652+0000","last_undegraded":"2024-09-17T16:02:19.442652+0000","last_fullsized":"2024-09-17T16:02:19.442652+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'33","last_scrub_stamp":"2024-09-17T16:02:02.109565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.109565+0000","objects_scrubbed":3,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.532828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.020286205000000002,"stat_sum":{"num_bytes":4063232,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":10525,"num_write":6,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.9","version":"81'47","reported_seq":306,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.458870+0000","last_change":"2024-09-17T16:02:19.459708+0000","last_active":"2024-09-17T16:02:19.458870+0000","last_peered":"2024-09-17T16:02:19.458870+0000","last_clean":"2024-09-17T16:02:19.458870+0000","last_became_active":"2024-09-17T16:01:52.347076+0000","last_became_peered":"2024-09-17T16:01:52.347076+0000","last_unstale":"2024-09-17T16:02:19.458870+0000","last_undegraded":"2024-09-17T16:02:19.458870+0000","last_fullsized":"2024-09-17T16:02:19.458870+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:02:01.251580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.251580+0000","objects_scrubbed":2,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:21.461356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.029814430999999999,"stat_sum":{"num_bytes":2949120,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":451,"num_write":4,"num_write_kb":1872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.a","version":"72'50","reported_seq":242,"reported_epoch":77,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:15.414487+0000","last_change":"2024-09-17T16:02:04.994580+0000","last_active":"2024-09-17T16:02:15.414487+0000","last_peered":"2024-09-17T16:02:15.414487+0000","last_clean":"2024-09-17T16:02:02.349037+0000","last_became_active":"2024-09-17T16:02:04.994580+0000","last_became_peered":"2024-09-17T16:02:04.994580+0000","last_unstale":"2024-09-17T16:02:15.414487+0000","last_undegraded":"2024-09-17T16:02:04.341211+0000","last_fullsized":"2024-09-17T16:02:04.340891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:58.849727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081311999999999,"stat_sum":{"num_bytes":10698752,"num_objects":9,"num_object_clones":7,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":17,"num_read_kb":20810,"num_write":23,"num_write_kb":10448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,2147483647,3,14,6,8],"acting":[12,4,2147483647,3,14,6,8],"avail_no_missing":["12(0)","3(3)","4(1)","6(5)","8(6)","14(4)"],"object_location_counts":[{"shards":"3(3),4(1),6(5),8(6),12(0),14(4)","objects":9}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.b","version":"69'19","reported_seq":232,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.430186+0000","last_change":"2024-09-17T16:02:04.347340+0000","last_active":"2024-09-17T16:02:19.430186+0000","last_peered":"2024-09-17T16:02:19.430186+0000","last_clean":"2024-09-17T16:02:02.337229+0000","last_became_active":"2024-09-17T16:02:04.347340+0000","last_became_peered":"2024-09-17T16:02:04.347340+0000","last_unstale":"2024-09-17T16:02:19.430186+0000","last_undegraded":"2024-09-17T16:02:04.340875+0000","last_fullsized":"2024-09-17T16:02:04.340555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.519072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010428939999999999,"stat_sum":{"num_bytes":1130496,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":1682,"num_write":9,"num_write_kb":1104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,2147483647],"acting":[13,1,11,15,6,4,2147483647],"avail_no_missing":["13(0)","1(1)","4(5)","6(4)","11(2)","15(3)"],"object_location_counts":[{"shards":"1(1),4(5),6(4),11(2),13(0),15(3)","objects":5}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.c","version":"81'49","reported_seq":281,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.446217+0000","last_change":"2024-09-17T16:02:19.447829+0000","last_active":"2024-09-17T16:02:19.446217+0000","last_peered":"2024-09-17T16:02:19.446217+0000","last_clean":"2024-09-17T16:02:19.446217+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:02:19.446217+0000","last_undegraded":"2024-09-17T16:02:19.446217+0000","last_fullsized":"2024-09-17T16:02:19.446217+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:52.914648+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017630505000000001,"stat_sum":{"num_bytes":1474560,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":11,"num_read_kb":2531,"num_write":32,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.d","version":"77'145","reported_seq":361,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-17T16:02:16.196591+0000","last_change":"2024-09-17T16:02:16.197507+0000","last_active":"2024-09-17T16:02:16.196591+0000","last_peered":"2024-09-17T16:02:16.196591+0000","last_clean":"2024-09-17T16:02:16.196591+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:02:16.196591+0000","last_undegraded":"2024-09-17T16:02:16.196591+0000","last_fullsized":"2024-09-17T16:02:16.196591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:37.552030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.782771624,"stat_sum":{"num_bytes":14778368,"num_objects":18,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":45,"num_read_kb":7039,"num_write":71,"num_write_kb":16304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"1"},{"start":"a","length":"1"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"}]},{"pgid":"3.e","version":"81'89","reported_seq":331,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.447755+0000","last_change":"2024-09-17T16:02:19.448576+0000","last_active":"2024-09-17T16:02:19.447755+0000","last_peered":"2024-09-17T16:02:19.447755+0000","last_clean":"2024-09-17T16:02:19.447755+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:02:19.447755+0000","last_undegraded":"2024-09-17T16:02:19.447755+0000","last_fullsized":"2024-09-17T16:02:19.447755+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:54.717396+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.019510643000000001,"stat_sum":{"num_bytes":8650752,"num_objects":11,"num_object_clones":7,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":31,"num_read_kb":11719,"num_write":68,"num_write_kb":6608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.f","version":"73'39","reported_seq":240,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.430912+0000","last_change":"2024-09-17T16:02:04.347090+0000","last_active":"2024-09-17T16:02:19.430912+0000","last_peered":"2024-09-17T16:02:19.430912+0000","last_clean":"2024-09-17T16:02:02.358329+0000","last_became_active":"2024-09-17T16:02:04.347090+0000","last_became_peered":"2024-09-17T16:02:04.347090+0000","last_unstale":"2024-09-17T16:02:19.430912+0000","last_undegraded":"2024-09-17T16:02:04.341882+0000","last_fullsized":"2024-09-17T16:02:04.341582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:39.704286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015713103999999999,"stat_sum":{"num_bytes":2605056,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":368,"num_write":36,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,2147483647,10,1],"acting":[0,2,14,11,2147483647,10,1],"avail_no_missing":["0(0)","1(6)","2(1)","10(5)","11(3)","14(2)"],"object_location_counts":[{"shards":"0(0),1(6),2(1),10(5),11(3),14(2)","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.10","version":"77'43","reported_seq":152,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.432923+0000","last_change":"2024-09-17T16:02:04.348295+0000","last_active":"2024-09-17T16:02:19.432923+0000","last_peered":"2024-09-17T16:02:19.432923+0000","last_clean":"2024-09-17T16:02:02.344595+0000","last_became_active":"2024-09-17T16:02:04.348295+0000","last_became_peered":"2024-09-17T16:02:04.348295+0000","last_unstale":"2024-09-17T16:02:19.432923+0000","last_undegraded":"2024-09-17T16:02:04.341600+0000","last_fullsized":"2024-09-17T16:02:04.341131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'31","last_scrub_stamp":"2024-09-17T16:01:58.728617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.728617+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:17.069070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.16505651299999999,"stat_sum":{"num_bytes":2916352,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,8,15,5,9,0],"acting":[2147483647,6,8,15,5,9,0],"avail_no_missing":["6(1)","0(6)","5(4)","8(2)","9(5)","15(3)"],"object_location_counts":[{"shards":"0(6),5(4),6(1),8(2),9(5),15(3)","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"76'18","reported_seq":129,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.435804+0000","last_change":"2024-09-17T16:02:04.349949+0000","last_active":"2024-09-17T16:02:19.435804+0000","last_peered":"2024-09-17T16:02:19.435804+0000","last_clean":"2024-09-17T16:02:02.345085+0000","last_became_active":"2024-09-17T16:02:04.349949+0000","last_became_peered":"2024-09-17T16:02:04.349949+0000","last_unstale":"2024-09-17T16:02:19.435804+0000","last_undegraded":"2024-09-17T16:02:04.342317+0000","last_fullsized":"2024-09-17T16:02:04.342004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.333371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.333371+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:10.093249+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011690380000000001,"stat_sum":{"num_bytes":3538944,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,2147483647,1],"acting":[10,4,0,3,5,2147483647,1],"avail_no_missing":["10(0)","0(2)","1(6)","3(3)","4(1)","5(4)"],"object_location_counts":[{"shards":"0(2),1(6),3(3),4(1),5(4),10(0)","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.12","version":"72'39","reported_seq":119,"reported_epoch":81,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:19.426361+0000","last_change":"2024-09-17T16:02:05.063706+0000","last_active":"2024-09-17T16:02:19.426361+0000","last_peered":"2024-09-17T16:02:19.426361+0000","last_clean":"2024-09-17T16:02:02.344908+0000","last_became_active":"2024-09-17T16:02:05.063706+0000","last_became_peered":"2024-09-17T16:02:05.063706+0000","last_unstale":"2024-09-17T16:02:19.426361+0000","last_undegraded":"2024-09-17T16:02:04.342739+0000","last_fullsized":"2024-09-17T16:02:04.342330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'32","last_scrub_stamp":"2024-09-17T16:02:01.034730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.034730+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:04.873493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0088005819999999995,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,6,12,2,2147483647],"acting":[15,8,5,6,12,2,2147483647],"avail_no_missing":["15(0)","2(5)","5(2)","6(3)","8(1)","12(4)"],"object_location_counts":[{"shards":"2(5),5(2),6(3),8(1),12(4),15(0)","objects":3}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.13","version":"81'36","reported_seq":307,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T16:02:19.441830+0000","last_change":"2024-09-17T16:02:19.444042+0000","last_active":"2024-09-17T16:02:19.441830+0000","last_peered":"2024-09-17T16:02:19.441830+0000","last_clean":"2024-09-17T16:02:19.441830+0000","last_became_active":"2024-09-17T16:01:52.366019+0000","last_became_peered":"2024-09-17T16:01:52.366019+0000","last_unstale":"2024-09-17T16:02:19.441830+0000","last_undegraded":"2024-09-17T16:02:19.441830+0000","last_fullsized":"2024-09-17T16:02:19.441830+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'16","last_scrub_stamp":"2024-09-17T16:02:00.060603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.060603+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:09.196592+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.017767330000000001,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":531,"num_write":1,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"e","length":"2"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":91029504,"num_objects":136,"num_object_clones":86,"num_object_copies":952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":63,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":9,"num_read":200,"num_read_kb":66985,"num_write":371,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":157540352,"data_stored":234688512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1098,"ondisk_log_size":1098,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39968,"kb_used_data":8780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331872,"statfs":{"total":96636764160,"available":96595836928,"internally_reserved":0,"allocated":8990720,"data_stored":12990920,"data_compressed":167579,"data_compressed_allocated":225280,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280528,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":8844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":9056256,"data_stored":11822002,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39996,"kb_used_data":12968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331844,"statfs":{"total":96636764160,"available":96595808256,"internally_reserved":0,"allocated":13279232,"data_stored":19198898,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313233,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43432,"kb_used_data":8084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94328408,"statfs":{"total":96636764160,"available":96592289792,"internally_reserved":0,"allocated":8278016,"data_stored":11037492,"data_compressed":173806,"data_compressed_allocated":233472,"data_compressed_original":466944,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48128,"kb_used_data":16936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323712,"statfs":{"total":96636764160,"available":96587481088,"internally_reserved":0,"allocated":17342464,"data_stored":25129906,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280528,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":13184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":13500416,"data_stored":20276146,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280528,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36108,"kb_used_data":9072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335732,"statfs":{"total":96636764160,"available":96599789568,"internally_reserved":0,"allocated":9289728,"data_stored":12899794,"data_compressed":187419,"data_compressed_allocated":274432,"data_compressed_original":987136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280528,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33824,"kb_used_data":6788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338016,"statfs":{"total":96636764160,"available":96602128384,"internally_reserved":0,"allocated":6950912,"data_stored":10355634,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280528,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45940,"kb_used_data":14748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325900,"statfs":{"total":96636764160,"available":96589721600,"internally_reserved":0,"allocated":15101952,"data_stored":22590386,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280528,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38760,"kb_used_data":11732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333080,"statfs":{"total":96636764160,"available":96597073920,"internally_reserved":0,"allocated":12013568,"data_stored":17572786,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280528,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38168,"kb_used_data":11140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333672,"statfs":{"total":96636764160,"available":96597680128,"internally_reserved":0,"allocated":11407360,"data_stored":16469080,"data_compressed":180029,"data_compressed_allocated":241664,"data_compressed_original":483328,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":5900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":6041600,"data_stored":8820876,"data_compressed":114770,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,5,6,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280528,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40052,"kb_used_data":13016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331788,"statfs":{"total":96636764160,"available":96595750912,"internally_reserved":0,"allocated":13328384,"data_stored":20079557,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280527,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31640,"kb_used_data":4612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340200,"statfs":{"total":96636764160,"available":96604364800,"internally_reserved":0,"allocated":4722688,"data_stored":5616581,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37624,"kb_used_data":6432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334216,"statfs":{"total":96636764160,"available":96598237184,"internally_reserved":0,"allocated":6586368,"data_stored":8631218,"data_compressed":176920,"data_compressed_allocated":237568,"data_compressed_original":475136,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280527,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44060,"kb_used_data":12864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327780,"statfs":{"total":96636764160,"available":96591646720,"internally_reserved":0,"allocated":13172736,"data_stored":19576274,"data_compressed":187419,"data_compressed_allocated":274432,"data_compressed_original":987136,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16318464,"data_stored":23826432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7380992,"data_stored":10391552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8323072,"data_stored":11513856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6041600,"data_stored":9699328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":14667776,"data_stored":22343680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11104256,"data_stored":16384000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":10485760,"data_stored":15802368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5378048,"data_stored":8372224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12414976,"data_stored":19091456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4169728,"data_stored":5787648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":6037504,"data_stored":8802304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12681216,"data_stored":19132416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":8118272,"data_stored":12365824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":12951552,"data_stored":19914752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":12845056,"data_stored":19152896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":8622080,"data_stored":12107776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:02:21.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 26 2024-09-17T16:02:21.187 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.188 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.195 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:21.193+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:21.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:21.296 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:21.294+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:21.298 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.298 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.307 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:21.397 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:21.395+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:21.409 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:21.409 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:21.410 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:21.410 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:21.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:21.433 INFO:tasks.rados.rados.0.smithi181.stdout:278: done (0 left) 2024-09-17T16:02:21.433 INFO:tasks.rados.rados.0.smithi181.stdout:279: snap_create 2024-09-17T16:02:21.476 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:21.477 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:21.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:21.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:02:21.498 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:21.496+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:21.527 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.528 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:21.599 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:21.598+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:21.681 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.682 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:21.700 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:21.699+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:21.798 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.798 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.801 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:21.799+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:21.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:21.902 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:21.901+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:21.913 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:21.913 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:21.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:22.003 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:22.002+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:22.030 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.031 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:22.104 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:22.102+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:22.146 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.146 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:22.205 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:22.203+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:22.261 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.261 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:22.315 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:22.313+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:22.377 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.377 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:22.406 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:22.404+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:22.429 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:02:22.430 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 3 pgp_num to 26 2024-09-17T16:02:22.432 INFO:tasks.rados.rados.0.smithi181.stdout:279: done (0 left) 2024-09-17T16:02:22.433 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 29 2024-09-17T16:02:22.433 INFO:tasks.rados.rados.0.smithi181.stdout:280: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-09-17T16:02:22.437 INFO:tasks.rados.rados.0.smithi181.stdout:280: writing smithi18136759-50 from 0 to 753664 tid 1 2024-09-17T16:02:22.440 INFO:tasks.rados.rados.0.smithi181.stdout:280: writing smithi18136759-50 from 753664 to 1441792 tid 2 2024-09-17T16:02:22.443 INFO:tasks.rados.rados.0.smithi181.stdout:280: writing smithi18136759-50 from 1441792 to 1916928 tid 3 2024-09-17T16:02:22.443 INFO:tasks.rados.rados.0.smithi181.stdout:281: rmattr oid 7 current snap is 29 2024-09-17T16:02:22.444 INFO:tasks.rados.rados.0.smithi181.stdout:281: done (1 left) 2024-09-17T16:02:22.444 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 2 current snap is 29 2024-09-17T16:02:22.444 INFO:tasks.rados.rados.0.smithi181.stdout:282: seq_num 115 ranges {32768=770048,802816=704512,1507328=458752} 2024-09-17T16:02:22.449 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18136759-2 from 32768 to 802816 tid 1 2024-09-17T16:02:22.452 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18136759-2 from 802816 to 1507328 tid 2 2024-09-17T16:02:22.455 INFO:tasks.rados.rados.0.smithi181.stdout:282: writing smithi18136759-2 from 1507328 to 1966080 tid 3 2024-09-17T16:02:22.455 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 22 current snap is 29 2024-09-17T16:02:22.455 INFO:tasks.rados.rados.0.smithi181.stdout:283: seq_num 116 ranges {0=786432,786432=458752,1245184=442368,1687552=262144} 2024-09-17T16:02:22.459 INFO:tasks.rados.rados.0.smithi181.stdout:283: writing smithi18136759-22 from 0 to 786432 tid 1 2024-09-17T16:02:22.462 INFO:tasks.rados.rados.0.smithi181.stdout:283: writing smithi18136759-22 from 786432 to 1245184 tid 2 2024-09-17T16:02:22.465 INFO:tasks.rados.rados.0.smithi181.stdout:283: writing smithi18136759-22 from 1245184 to 1687552 tid 3 2024-09-17T16:02:22.466 INFO:tasks.rados.rados.0.smithi181.stdout:283: writing smithi18136759-22 from 1687552 to 1949696 tid 4 2024-09-17T16:02:22.466 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 32 current snap is 29 2024-09-17T16:02:22.467 INFO:tasks.rados.rados.0.smithi181.stdout:284: seq_num 117 ranges {0=425984,425984=606208,1032192=540672,1572864=409600} 2024-09-17T16:02:22.469 INFO:tasks.rados.rados.0.smithi181.stdout:284: writing smithi18136759-32 from 0 to 425984 tid 1 2024-09-17T16:02:22.473 INFO:tasks.rados.rados.0.smithi181.stdout:284: writing smithi18136759-32 from 425984 to 1032192 tid 2 2024-09-17T16:02:22.476 INFO:tasks.rados.rados.0.smithi181.stdout:284: writing smithi18136759-32 from 1032192 to 1572864 tid 3 2024-09-17T16:02:22.478 INFO:tasks.rados.rados.0.smithi181.stdout:284: writing smithi18136759-32 from 1572864 to 1982464 tid 4 2024-09-17T16:02:22.478 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 9 current snap is 29 2024-09-17T16:02:22.478 INFO:tasks.rados.rados.0.smithi181.stdout:285: seq_num 118 ranges {0=442368,442368=753664,1196032=655360,1851392=147456} 2024-09-17T16:02:22.480 INFO:tasks.rados.rados.0.smithi181.stdout:285: writing smithi18136759-9 from 0 to 442368 tid 1 2024-09-17T16:02:22.485 INFO:tasks.rados.rados.0.smithi181.stdout:285: writing smithi18136759-9 from 442368 to 1196032 tid 2 2024-09-17T16:02:22.488 INFO:tasks.rados.rados.0.smithi181.stdout:285: writing smithi18136759-9 from 1196032 to 1851392 tid 3 2024-09-17T16:02:22.490 INFO:tasks.rados.rados.0.smithi181.stdout:285: writing smithi18136759-9 from 1851392 to 1998848 tid 4 2024-09-17T16:02:22.490 INFO:tasks.rados.rados.0.smithi181.stdout:286: rmattr oid 41 current snap is 29 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:286: done (5 left) 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 24 current snap is 29 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: finishing write tid 1 to smithi18136759-50 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: finishing write tid 2 to smithi18136759-50 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: finishing write tid 3 to smithi18136759-50 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: finishing write tid 4 to smithi18136759-50 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: finishing write tid 5 to smithi18136759-50 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: oid 50 updating version 0 to 100 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: oid 50 version 100 is already newer than 98 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: oid 50 updating version 100 to 101 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: oid 50 version 101 is already newer than 99 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 101 (ObjNum 114 snap 29 seq_num 114) dirty exists 2024-09-17T16:02:22.491 INFO:tasks.rados.rados.0.smithi181.stdout:280: left oid 50 (ObjNum 114 snap 29 seq_num 114) 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 1 to smithi18136759-2 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 2 to smithi18136759-2 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 3 to smithi18136759-2 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 4 to smithi18136759-2 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 1 to smithi18136759-22 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: finishing write tid 5 to smithi18136759-2 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 2 updating version 0 to 24 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 2 version 24 is already newer than 21 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 2 version 24 is already newer than 23 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: oid 2 version 24 is already newer than 22 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 24 (ObjNum 115 snap 29 seq_num 115) dirty exists 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:282: left oid 2 (ObjNum 115 snap 29 seq_num 115) 2024-09-17T16:02:22.492 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 2 to smithi18136759-22 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 3 to smithi18136759-22 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 4 to smithi18136759-22 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 5 to smithi18136759-22 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: finishing write tid 6 to smithi18136759-22 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: oid 22 updating version 0 to 17 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: oid 22 updating version 17 to 18 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: oid 22 updating version 18 to 19 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: oid 22 updating version 19 to 20 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: oid 22 version 20 is already newer than 16 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 20 (ObjNum 116 snap 29 seq_num 116) dirty exists 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:283: left oid 22 (ObjNum 116 snap 29 seq_num 116) 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 1 to smithi18136759-32 2024-09-17T16:02:22.493 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 2 to smithi18136759-32 2024-09-17T16:02:22.494 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 3 to smithi18136759-32 2024-09-17T16:02:22.494 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 4 to smithi18136759-32 2024-09-17T16:02:22.494 INFO:tasks.rados.rados.0.smithi181.stdout:287: seq_num 119 ranges {1736704=458752,2195456=507904,2703360=507904,3211264=458752,3670016=81920} 2024-09-17T16:02:22.501 INFO:tasks.rados.rados.0.smithi181.stdout:287: writing smithi18136759-24 from 1736704 to 2195456 tid 1 2024-09-17T16:02:22.504 INFO:tasks.rados.rados.0.smithi181.stdout:287: writing smithi18136759-24 from 2195456 to 2703360 tid 2 2024-09-17T16:02:22.507 INFO:tasks.rados.rados.0.smithi181.stdout:287: writing smithi18136759-24 from 2703360 to 3211264 tid 3 2024-09-17T16:02:22.507 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:22.505+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:22.509 INFO:tasks.rados.rados.0.smithi181.stdout:287: writing smithi18136759-24 from 3211264 to 3670016 tid 4 2024-09-17T16:02:22.510 INFO:tasks.rados.rados.0.smithi181.stdout:287: writing smithi18136759-24 from 3670016 to 3751936 tid 5 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:280: done (5 left) 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:282: done (4 left) 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:283: done (3 left) 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:288: rollback oid 36 current snap is 29 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 36 to 20 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 5 to smithi18136759-32 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:289: read oid 34 snap 14 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:289: expect (ObjNum 72 snap 13 seq_num 72) 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:290: rmattr oid 48 current snap is 29 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 1 to smithi18136759-9 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: finishing write tid 6 to smithi18136759-32 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: oid 32 updating version 0 to 53 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: oid 32 version 53 is already newer than 52 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: oid 32 version 53 is already newer than 51 2024-09-17T16:02:22.511 INFO:tasks.rados.rados.0.smithi181.stdout:284: oid 32 version 53 is already newer than 50 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:284: oid 32 version 53 is already newer than 49 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 32 v 53 (ObjNum 117 snap 29 seq_num 117) dirty exists 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:284: left oid 32 (ObjNum 117 snap 29 seq_num 117) 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 2 to smithi18136759-9 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 3 to smithi18136759-9 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:284: done (5 left) 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:291: rmattr oid 40 current snap is 29 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:291: done (5 left) 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 33 current snap is 29 2024-09-17T16:02:22.512 INFO:tasks.rados.rados.0.smithi181.stdout:292: seq_num 120 ranges {1884160=475136,2359296=655360,3014656=622592,3637248=278528} 2024-09-17T16:02:22.522 INFO:tasks.rados.rados.0.smithi181.stdout:292: writing smithi18136759-33 from 1884160 to 2359296 tid 1 2024-09-17T16:02:22.526 INFO:tasks.rados.rados.0.smithi181.stdout:292: writing smithi18136759-33 from 2359296 to 3014656 tid 2 2024-09-17T16:02:22.529 INFO:tasks.rados.rados.0.smithi181.stdout:292: writing smithi18136759-33 from 3014656 to 3637248 tid 3 2024-09-17T16:02:22.531 INFO:tasks.rados.rados.0.smithi181.stdout:292: writing smithi18136759-33 from 3637248 to 3915776 tid 4 2024-09-17T16:02:22.531 INFO:tasks.rados.rados.0.smithi181.stdout:293: rollback oid 38 current snap is 29 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 38 to 26 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 4 to smithi18136759-9 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 5 to smithi18136759-9 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: finishing write tid 6 to smithi18136759-9 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: oid 9 updating version 0 to 54 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: oid 9 updating version 54 to 55 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: oid 9 version 55 is already newer than 53 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: oid 9 updating version 55 to 56 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: oid 9 version 56 is already newer than 52 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 56 (ObjNum 118 snap 29 seq_num 118) dirty exists 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: left oid 9 (ObjNum 118 snap 29 seq_num 118) 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:285: done (6 left) 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 47 current snap is 29 2024-09-17T16:02:22.532 INFO:tasks.rados.rados.0.smithi181.stdout:294: seq_num 121 ranges {786432=491520,1277952=802816,2080768=720896,2801664=32768} 2024-09-17T16:02:22.538 INFO:tasks.rados.rados.0.smithi181.stdout:294: writing smithi18136759-47 from 786432 to 1277952 tid 1 2024-09-17T16:02:22.543 INFO:tasks.rados.rados.0.smithi181.stdout:294: writing smithi18136759-47 from 1277952 to 2080768 tid 2 2024-09-17T16:02:22.547 INFO:tasks.rados.rados.0.smithi181.stdout:294: writing smithi18136759-47 from 2080768 to 2801664 tid 3 2024-09-17T16:02:22.547 INFO:tasks.rados.rados.0.smithi181.stdout:294: writing smithi18136759-47 from 2801664 to 2834432 tid 4 2024-09-17T16:02:22.547 INFO:tasks.rados.rados.0.smithi181.stdout:295: rollback oid 16 current snap is 29 2024-09-17T16:02:22.547 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 16 to 10 2024-09-17T16:02:22.547 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing rollback tid 1 to smithi18136759-36 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:288: finishing rollback tid 2 to smithi18136759-36 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 103 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 1 to smithi18136759-24 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 2 to smithi18136759-24 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 3 to smithi18136759-24 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 48 v 105 (ObjNum 64 snap 9 seq_num 64) dirty exists 2024-09-17T16:02:22.548 INFO:tasks.rados.rados.0.smithi181.stdout:289: expect (ObjNum 72 snap 13 seq_num 72) 2024-09-17T16:02:22.556 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.556 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.556 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1722812 2024-09-17T16:02:22.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:22.566 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1706474 2024-09-17T16:02:22.572 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 990872 2024-09-17T16:02:22.579 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 4 to smithi18136759-24 2024-09-17T16:02:22.579 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 5 to smithi18136759-24 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 6 to smithi18136759-24 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: finishing write tid 7 to smithi18136759-24 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 updating version 0 to 185 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 version 185 is already newer than 181 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 version 185 is already newer than 183 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 version 185 is already newer than 184 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 updating version 185 to 186 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: oid 24 version 186 is already newer than 182 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 24 v 186 (ObjNum 119 snap 29 seq_num 119) dirty exists 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:287: left oid 24 (ObjNum 119 snap 29 seq_num 119) 2024-09-17T16:02:22.580 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 1 to smithi18136759-33 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:287: done (7 left) 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:288: done (6 left) 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:289: done (5 left) 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:290: done (4 left) 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 29 current snap is 29 2024-09-17T16:02:22.581 INFO:tasks.rados.rados.0.smithi181.stdout:296: seq_num 122 ranges {1769472=507904,2277376=425984,2703360=442368,3145728=540672,3686400=147456} 2024-09-17T16:02:22.591 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18136759-29 from 1769472 to 2277376 tid 1 2024-09-17T16:02:22.594 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18136759-29 from 2277376 to 2703360 tid 2 2024-09-17T16:02:22.596 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18136759-29 from 2703360 to 3145728 tid 3 2024-09-17T16:02:22.599 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18136759-29 from 3145728 to 3686400 tid 4 2024-09-17T16:02:22.600 INFO:tasks.rados.rados.0.smithi181.stdout:296: writing smithi18136759-29 from 3686400 to 3833856 tid 5 2024-09-17T16:02:22.600 INFO:tasks.rados.rados.0.smithi181.stdout:297: delete oid 20 current snap is 29 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:293: finishing rollback tid 0 to smithi18136759-38 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 2 to smithi18136759-33 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:293: finishing rollback tid 1 to smithi18136759-38 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 3 to smithi18136759-33 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 4 to smithi18136759-33 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 1 to smithi18136759-47 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing rollback tid 0 to smithi18136759-16 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 2 to smithi18136759-47 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing rollback tid 1 to smithi18136759-16 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 3 to smithi18136759-47 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 4 to smithi18136759-47 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:295: finishing rollback tid 2 to smithi18136759-16 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 34 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-17T16:02:22.601 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 1 to smithi18136759-29 2024-09-17T16:02:22.603 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 2 to smithi18136759-29 2024-09-17T16:02:22.604 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 3 to smithi18136759-29 2024-09-17T16:02:22.605 INFO:tasks.rados.rados.0.smithi181.stdout:295: done (5 left) 2024-09-17T16:02:22.605 INFO:tasks.rados.rados.0.smithi181.stdout:297: done (4 left) 2024-09-17T16:02:22.605 INFO:tasks.rados.rados.0.smithi181.stdout:298: snap_create 2024-09-17T16:02:22.605 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 4 to smithi18136759-29 2024-09-17T16:02:22.605 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 5 to smithi18136759-29 2024-09-17T16:02:22.606 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 6 to smithi18136759-29 2024-09-17T16:02:22.607 INFO:tasks.rados.rados.0.smithi181.stdout:296: finishing write tid 7 to smithi18136759-29 2024-09-17T16:02:22.607 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 updating version 0 to 193 2024-09-17T16:02:22.607 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 version 193 is already newer than 192 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 version 193 is already newer than 190 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 version 193 is already newer than 189 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 version 193 is already newer than 188 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:296: oid 29 version 193 is already newer than 191 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 29 v 193 (ObjNum 122 snap 29 seq_num 122) dirty exists 2024-09-17T16:02:22.608 INFO:tasks.rados.rados.0.smithi181.stdout:296: left oid 29 (ObjNum 122 snap 29 seq_num 122) 2024-09-17T16:02:22.609 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:22.606+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:22.708 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:22.706+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:22.712 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.712 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:22.810 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:22.808+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:22.828 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.828 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:22.911 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:22.909+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:22.944 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:22.945 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:22.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:23.012 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:23.010+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:23.047 INFO:tasks.rados.rados.0.smithi181.stdout:293: finishing rollback tid 2 to smithi18136759-38 2024-09-17T16:02:23.048 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 38 v 39 (ObjNum 97 snap 0 seq_num 1475414736) dirty exists 2024-09-17T16:02:23.101 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:23.101 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:23.107 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 5 to smithi18136759-33 2024-09-17T16:02:23.108 INFO:tasks.rados.rados.0.smithi181.stdout:292: finishing write tid 6 to smithi18136759-33 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: oid 33 updating version 0 to 42 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: oid 33 updating version 42 to 43 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: oid 33 version 43 is already newer than 41 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: oid 33 version 43 is already newer than 40 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: oid 33 updating version 43 to 44 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 33 v 44 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-09-17T16:02:23.109 INFO:tasks.rados.rados.0.smithi181.stdout:292: left oid 33 (ObjNum 120 snap 29 seq_num 120) 2024-09-17T16:02:23.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:23.112 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:23.110+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:23.213 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:23.211+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:23.217 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:23.217 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:23.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:23.280 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 5 to smithi18136759-47 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: finishing write tid 6 to smithi18136759-47 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: oid 47 updating version 0 to 29 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: oid 47 version 29 is already newer than 28 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: oid 47 version 29 is already newer than 27 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: oid 47 updating version 29 to 30 2024-09-17T16:02:23.283 INFO:tasks.rados.rados.0.smithi181.stdout:294: oid 47 version 30 is already newer than 26 2024-09-17T16:02:23.284 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 30 (ObjNum 121 snap 29 seq_num 121) dirty exists 2024-09-17T16:02:23.284 INFO:tasks.rados.rados.0.smithi181.stdout:294: left oid 47 (ObjNum 121 snap 29 seq_num 121) 2024-09-17T16:02:23.314 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:23.312+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:23.347 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:23.348 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:23.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:23.415 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:23.413+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:292: done (4 left) 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:293: done (3 left) 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:294: done (2 left) 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:296: done (1 left) 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:298: done (0 left) 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:299: copy_from oid 18 from oid 36 current snap is 30 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:300: rollback oid 22 current snap is 30 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 22 to 22 2024-09-17T16:02:23.458 INFO:tasks.rados.rados.0.smithi181.stdout:301: snap_create 2024-09-17T16:02:23.464 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 0 to smithi18136759-22 2024-09-17T16:02:23.465 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 1 to smithi18136759-22 2024-09-17T16:02:23.469 INFO:tasks.rados.rados.0.smithi181.stdout:299: finishing copy_from racing read to smithi18136759-18 2024-09-17T16:02:23.470 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:23.470 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:23.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:23.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:23.515 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:23.514+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:23.616 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:23.614+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:23.617 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:23.618 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:23.626 INFO:tasks.rados.rados.0.smithi181.stdout:300: finishing rollback tid 2 to smithi18136759-22 2024-09-17T16:02:23.626 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 22 v 24 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-17T16:02:23.627 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:23.627 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:23.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:23.640 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:23.640 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:23.662 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:23.662 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:23.684 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:23.684 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:23.704 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:23.704 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:23.717 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:23.715+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:23.730 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:23.743 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:23.744 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:23.744 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:23.744 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:23.752 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:23.752 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:23.753 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:23.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:23.772 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:23.773 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:23.793 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:23.794 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:23.815 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:23.815 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:23.818 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:23.815+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:23.837 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:23.837 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:23.857 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:23.858 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:23.860 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:23.860 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:23.860 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:23.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:23.879 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:23.879 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:23.900 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:23.901 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:23.899+0000 7fb0e0827640 0 -- 172.21.15.6:0/409888963 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fb0dc21d4a0 msgr2=0x7fb0dc21f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:23.918 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:23.916+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:23.977 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:23.977 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:23.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:23.977 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.045772522, 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:23.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:02:23.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:23.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:23.982 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.055103531, 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026452619999999998, 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.14e-07 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919991+0000", 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068760399999999997 2024-09-17T16:02:23.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920025+0000", 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4292000000000003e-05 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920043+0000", 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8028000000000001e-05 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:23.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920118+0000", 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4583000000000001e-05 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920129+0000", 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1105e-05 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921921+0000", 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017915590000000001 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921949+0000", 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7977e-05 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.049020706, 2024-09-17T16:02:23.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:23.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:02:23.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.04219509, 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:23.992 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:23.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:02:23.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.996 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.622870681999999, 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017347922000000002, 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.997 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351537+0000", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351558+0000", 2024-09-17T16:02:23.998 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1399e-05 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367847+0000", 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289428000000002 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367858+0000", 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0835e-05 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:23.999 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367938+0000", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0104000000000004e-05 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367950+0000", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1153e-05 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368846+0000", 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089654899999999996 2024-09-17T16:02:24.000 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368884+0000", 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8075999999999999e-05 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.581276574, 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047653560000000001, 2024-09-17T16:02:24.001 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.002 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393131+0000", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9600000000000008e-07 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394022+0000", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089089400000000004 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394066+0000", 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4193000000000002e-05 2024-09-17T16:02:24.003 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394089+0000", 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2608000000000001e-05 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394165+0000", 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5638999999999999e-05 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.004 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394175+0000", 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0277e-05 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397846+0000", 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036707889999999998 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397896+0000", 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9960000000000003e-05 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.005 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=64'34, trim_to=0'0, roll_forward_to=62'32))", 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.637126149, 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035966639999999999, 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:24.006 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.04e-07 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.007 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337334+0000", 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3352e-05 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337390+0000", 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5319999999999999e-05 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337400+0000", 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:02:24.008 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337509+0000", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108571 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337523+0000", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4211e-05 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340822+0000", 2024-09-17T16:02:24.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032988689999999998 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340877+0000", 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5481e-05 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.635116378999999, 2024-09-17T16:02:24.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027044159999999999, 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:24.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339292+0000", 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.187e-06 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339317+0000", 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5355e-05 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339357+0000", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0244000000000003e-05 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339368+0000", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0512e-05 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339491+0000", 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012284700000000001 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339502+0000", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0848e-05 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341950+0000", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002448703 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341995+0000", 2024-09-17T16:02:24.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4719999999999999e-05 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=42, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=64'34))", 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.629345896, 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027339, 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:24.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.28e-07 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345069+0000", 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6920000000000002e-06 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345381+0000", 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031179599999999998 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345390+0000", 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5810000000000003e-06 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345492+0000", 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101902 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345506+0000", 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3395e-05 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347746+0000", 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002240845 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347795+0000", 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8461000000000002e-05 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s2 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:24.019 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.629094423000002, 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035907539999999998, 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345313+0000", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345362+0000", 2024-09-17T16:02:24.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8804000000000002e-05 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345410+0000", 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8133999999999997e-05 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345419+0000", 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6039999999999999e-06 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345499+0000", 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0173000000000006e-05 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345510+0000", 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348833+0000", 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033223749999999998 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348903+0000", 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0499000000000004e-05 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 21.627697137999998, 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034797959999999998, 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.024 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346711+0000", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6600000000000005e-07 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346735+0000", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4652e-05 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346773+0000", 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7855999999999998e-05 2024-09-17T16:02:24.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346782+0000", 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5599999999999994e-06 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346854+0000", 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2512e-05 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346865+0000", 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0957e-05 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350146+0000", 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032811709999999998 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350190+0000", 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3222000000000001e-05 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.028 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 76/32 ECSubWrite(tid=207, reqid=client.4560.0:597, at_version=76'132, trim_to=0'0, roll_forward_to=75'130))", 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 9.5652613760000005, 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045991310000000002, 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:24.030 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410426+0000", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012808170000000001 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410463+0000", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6449000000000003e-05 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410471+0000", 2024-09-17T16:02:24.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8420000000000005e-06 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410556+0000", 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5426999999999994e-05 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410574+0000", 2024-09-17T16:02:24.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7306999999999999e-05 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413700+0000", 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031262460000000001 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413745+0000", 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4910000000000002e-05 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5424386779999999, 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036738819999999998, 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:24.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431969+0000", 2024-09-17T16:02:24.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3500000000000002e-07 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431996+0000", 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7461e-05 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432037+0000", 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0552000000000001e-05 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432059+0000", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2345e-05 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432134+0000", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5159000000000003e-05 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432145+0000", 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0875e-05 2024-09-17T16:02:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435607+0000", 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034618510000000002 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435642+0000", 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5203999999999997e-05 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=225, reqid=client.4560.0:627, at_version=79'152, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5413924359999998, 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003614744, 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433015+0000", 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433165+0000", 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149612 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433313+0000", 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014852600000000001 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433322+0000", 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4770000000000003e-06 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433429+0000", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010759099999999999 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433441+0000", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1323e-05 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436598+0000", 2024-09-17T16:02:24.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031569749999999998 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436629+0000", 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1473000000000002e-05 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.5412354490000002, 2024-09-17T16:02:24.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034963870000000001, 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:24.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433181+0000", 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7489999999999993e-06 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433276+0000", 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4340999999999997e-05 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433357+0000", 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1524000000000001e-05 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433364+0000", 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5710000000000003e-06 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434090+0000", 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072590199999999995 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434157+0000", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7516999999999997e-05 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436645+0000", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024882789999999999 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436668+0000", 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2504e-05 2024-09-17T16:02:24.047 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 80/32 ECSubWrite(tid=233, reqid=client.4560.0:664, at_version=80'156, trim_to=0'0, roll_forward_to=79'154))", 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.5275433559999998, 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030183900000000001, 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:24.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1899999999999999e-07 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447421+0000", 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055686899999999998 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447452+0000", 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1937999999999998e-05 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447462+0000", 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447533+0000", 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1507999999999994e-05 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447546+0000", 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3174e-05 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.449848+0000", 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002301403 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.449882+0000", 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4069999999999997e-05 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=240, reqid=osd.12.0:240, at_version=81'161, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5448777620000005, 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003961633, 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:24.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429579+0000", 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9345999999999998e-05 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430294+0000", 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071481400000000003 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430305+0000", 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1618e-05 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430487+0000", 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018190799999999999 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430503+0000", 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.643e-05 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433445+0000", 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029419300000000002 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433491+0000", 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5348e-05 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=241, reqid=osd.12.0:241, at_version=81'163, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:24.057 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5448245360000001, 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0053980119999999998, 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:24.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429583+0000", 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2399999999999999e-07 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429601+0000", 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8195000000000001e-05 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430343+0000", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074222800000000005 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430351+0000", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5329999999999997e-06 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431146+0000", 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079573700000000003 2024-09-17T16:02:24.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431157+0000", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0696999999999999e-05 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434934+0000", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037763610000000002 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434980+0000", 2024-09-17T16:02:24.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6836999999999999e-05 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5431017579999997, 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036787539999999998, 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.29e-07 2024-09-17T16:02:24.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431329+0000", 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3555000000000001e-05 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431411+0000", 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1885999999999995e-05 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431421+0000", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0139999999999999e-05 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431508+0000", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7016999999999997e-05 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431520+0000", 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2160999999999999e-05 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434938+0000", 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034178939999999999 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434984+0000", 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5871999999999997e-05 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.066 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.5430752879999998, 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040176439999999999, 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431352+0000", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0449e-05 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431581+0000", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022874899999999999 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431589+0000", 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0299999999999994e-06 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.432989+0000", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014002800000000001 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433001+0000", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2004999999999999e-05 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435321+0000", 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002319883 2024-09-17T16:02:24.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435349+0000", 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8008e-05 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.071 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:24.073 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:24.070+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:24.124 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:24.124 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:24.128 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:24.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:24.174 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:24.172+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:24.220 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.221 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:24.221 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:24.221 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:24.275 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:24.273+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:24.337 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.337 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:24.337 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:24.337 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:24.337 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:24.346 INFO:tasks.rados.rados.0.smithi181.stdout:299: finishing copy_from to smithi18136759-18 2024-09-17T16:02:24.346 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 18 v 103 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-17T16:02:24.375 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:24.373+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:24.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:24.455 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.595708809999998, 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:02:24.457 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.458 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:02:24.459 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.592746318000003, 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.460 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.461 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.462 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.463 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.589115988000003, 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.464 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.465 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.466 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.467 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.468 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.564955163, 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.470 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:02:24.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.560403203, 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.473 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:24.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:02:24.475 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.477 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.526000263999997, 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019463340000000001, 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9200000000000002e-07 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037e-05 2024-09-17T16:02:24.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926741+0000", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0134999999999999e-05 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926751+0000", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926822+0000", 2024-09-17T16:02:24.480 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0667000000000006e-05 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926854+0000", 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928578+0000", 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017237979999999999 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.481 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928603+0000", 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5769e-05 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.534224494, 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020717940000000001, 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.483 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6899999999999999e-07 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918457+0000", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918505+0000", 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7648000000000002e-05 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8609999999999993e-06 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918721+0000", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020663600000000001 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918732+0000", 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1058e-05 2024-09-17T16:02:24.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920477+0000", 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017447840000000001 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920505+0000", 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7815000000000001e-05 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.528388591000002, 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020194549999999999, 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:24.487 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1600000000000002e-07 2024-09-17T16:02:24.488 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924293+0000", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924390+0000", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6893999999999998e-05 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924399+0000", 2024-09-17T16:02:24.489 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924465+0000", 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309000000000005e-05 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924474+0000", 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5289999999999997e-06 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.490 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926235+0000", 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001760529 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926288+0000", 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3263000000000001e-05 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s6 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:24.491 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.079656137000001, 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003094585, 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.492 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.61e-07 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373230+0000", 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022871100000000001 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373269+0000", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8470999999999998e-05 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373278+0000", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8449999999999998e-06 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373359+0000", 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0401999999999994e-05 2024-09-17T16:02:24.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373368+0000", 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1919999999999994e-06 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376052+0000", 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026842319999999999 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.495 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376096+0000", 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3571e-05 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 31.055831153, 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020885859999999999, 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.496 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:24.497 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397062+0000", 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236006 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397106+0000", 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3995000000000001e-05 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.498 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397119+0000", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397201+0000", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2498999999999997e-05 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397214+0000", 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2819999999999999e-05 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398870+0000", 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016554829999999999 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398915+0000", 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5083000000000002e-05 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.500 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.109987228000001, 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038094829999999998, 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342671+0000", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342695+0000", 2024-09-17T16:02:24.502 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4366999999999999e-05 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343531+0000", 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083620199999999997 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343542+0000", 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0672e-05 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.503 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343828+0000", 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000285834 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343846+0000", 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8471999999999999e-05 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346440+0000", 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025931499999999998 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.504 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346480+0000", 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0064000000000001e-05 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.106917106000001, 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027362839999999999, 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.505 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.506 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1799999999999999e-07 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0065999999999999e-05 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345845+0000", 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4876999999999996e-05 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.507 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345856+0000", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1073000000000001e-05 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345935+0000", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9010000000000004e-05 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345947+0000", 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1708999999999999e-05 2024-09-17T16:02:24.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348437+0000", 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024899560000000002 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348477+0000", 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9375000000000002e-05 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.509 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s5 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.106876482000001, 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034936559999999999, 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.510 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.92e-07 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345815+0000", 2024-09-17T16:02:24.511 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3519999999999998e-05 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345868+0000", 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3248000000000002e-05 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345877+0000", 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7970000000000006e-06 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.512 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346334+0000", 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045731999999999999 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346345+0000", 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0854e-05 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349226+0000", 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028807569999999998 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.513 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349275+0000", 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8968000000000003e-05 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.106025209999999, 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002590527, 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:24.515 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.73e-07 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346659+0000", 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7055000000000001e-05 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346738+0000", 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8388999999999998e-05 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.516 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346746+0000", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4090000000000003e-06 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346817+0000", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0895999999999993e-05 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346826+0000", 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349183+0000", 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023570639999999999 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349223+0000", 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607999999999998e-05 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.518 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.105995483000001, 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037577729999999998, 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346683+0000", 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0866999999999999e-05 2024-09-17T16:02:24.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347481+0000", 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079794800000000004 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347491+0000", 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.521 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348761+0000", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001270188 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348771+0000", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.815e-06 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350377+0000", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016056499999999999 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.522 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350420+0000", 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2944000000000001e-05 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.101693049000001, 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027021179999999999, 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.523 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.524 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350965+0000", 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.99e-07 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350989+0000", 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4145999999999999e-05 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351039+0000", 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9858000000000003e-05 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.525 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351051+0000", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2729e-05 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351123+0000", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1680000000000005e-05 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351135+0000", 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2167000000000001e-05 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353616+0000", 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002481287 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353666+0000", 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0052000000000001e-05 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.527 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0071035349999997, 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026861049999999998, 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445555+0000", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9299999999999997e-07 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446015+0000", 2024-09-17T16:02:24.529 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046028900000000001 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446034+0000", 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9279e-05 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446041+0000", 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.530 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446100+0000", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8473000000000001e-05 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446107+0000", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9319999999999999e-06 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448192+0000", 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020857570000000001 2024-09-17T16:02:24.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448240+0000", 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7469000000000002e-05 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 81/53 ECSubWrite(tid=47, reqid=osd.8.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=81'22))", 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0057778239999999, 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002323982, 2024-09-17T16:02:24.532 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446881+0000", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.232e-06 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446896+0000", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4976e-05 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446962+0000", 2024-09-17T16:02:24.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6397999999999997e-05 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446971+0000", 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2609999999999999e-06 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447062+0000", 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0257e-05 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447074+0000", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2108e-05 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449155+0000", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020816559999999999 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449204+0000", 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8093999999999997e-05 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 81/32 ECSubWrite(tid=102, reqid=osd.0.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=81'49))", 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0057265429999998, 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022743210000000002, 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446942+0000", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0988999999999999e-05 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447265+0000", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032216900000000003 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447274+0000", 2024-09-17T16:02:24.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4800000000000007e-06 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447363+0000", 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9007999999999999e-05 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447374+0000", 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1121000000000001e-05 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.540 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449154+0000", 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001779997 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449205+0000", 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0948000000000001e-05 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:24.541 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.0038054199999999, 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023273109999999999, 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.542 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448853+0000", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449276+0000", 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000422757 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.543 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449299+0000", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3098000000000001e-05 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449308+0000", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0159999999999997e-06 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449374+0000", 2024-09-17T16:02:24.544 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6029000000000001e-05 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449385+0000", 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1019000000000001e-05 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451144+0000", 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001759287 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4966999999999999e-05 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.546 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:299: done (2 left) 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:300: done (1 left) 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:301: done (0 left) 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:302: rollback oid 10 current snap is 31 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 10 to 14 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:303: read oid 22 snap 10 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:303: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:304: read oid 32 snap 14 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:304: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 44 current snap is 31 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:305: seq_num 124 ranges {3227648=540672,3768320=720896,4489216=786432,5275648=49152} 2024-09-17T16:02:24.550 INFO:tasks.rados.rados.0.smithi181.stdout:305: writing smithi18136759-44 from 3227648 to 3768320 tid 1 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:305: writing smithi18136759-44 from 3768320 to 4489216 tid 2 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:305: writing smithi18136759-44 from 4489216 to 5275648 tid 3 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:305: writing smithi18136759-44 from 5275648 to 5324800 tid 4 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:306: delete oid 14 current snap is 31 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:302: finishing rollback tid 0 to smithi18136759-10 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:302: finishing rollback tid 1 to smithi18136759-10 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:303: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 96526 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:302: finishing rollback tid 2 to smithi18136759-10 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 10 v 199 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 1 to smithi18136759-44 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 2 to smithi18136759-44 2024-09-17T16:02:24.551 INFO:tasks.rados.rados.0.smithi181.stdout:302: done (4 left) 2024-09-17T16:02:24.552 INFO:tasks.rados.rados.0.smithi181.stdout:303: done (3 left) 2024-09-17T16:02:24.552 INFO:tasks.rados.rados.0.smithi181.stdout:306: done (2 left) 2024-09-17T16:02:24.552 INFO:tasks.rados.rados.0.smithi181.stdout:307: snap_create 2024-09-17T16:02:24.552 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 3 to smithi18136759-44 2024-09-17T16:02:24.552 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:24.546+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:24.648 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:24.647+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:24.659 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.659 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:24.660 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:24.660 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:24.750 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:24.747+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:24.775 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.775 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:24.775 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:24.775 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:24.775 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:24.851 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:24.849+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:24.862 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 4 to smithi18136759-44 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:6 3.bs0 3:d4004f6b:::smithi18136759-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.036808944999997, 2024-09-17T16:02:24.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013441952, 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 6 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:24.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852367+0000", 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852370+0000", 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852376+0000", 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.331e-06 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852381+0000", 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9169999999999997e-06 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026562230000000001 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855122+0000", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4809999999999996e-05 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864102+0000", 2024-09-17T16:02:24.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089801619999999999 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864209+0000", 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107002 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865809+0000", 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0015998270000000001 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:20 3.1s0 3:8347559a:::smithi18136759-7:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.036459985999997, 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.01273402, 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 20 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852716+0000", 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852717+0000", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.088e-06 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852722+0000", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852733+0000", 2024-09-17T16:02:24.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0414999999999999e-05 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852741+0000", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1280000000000008e-06 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.854959+0000", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002218603 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855051+0000", 2024-09-17T16:02:24.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1361999999999997e-05 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855580+0000", 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00052929699999999997 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856734+0000", 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011540089999999999 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865354+0000", 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0086203630000000007 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865450+0000", 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5616000000000001e-05 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.036419854999998, 2024-09-17T16:02:24.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:24.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.908 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.023742560999999, 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:24.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:02:24.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:02:24.913 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.023469319999997, 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.914 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:02:24.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:02:24.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.020603538000003, 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.919 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:02:24.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:02:24.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.020551611000002, 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.925 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:02:24.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.000010844999998, 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.930 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:02:24.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.999927671000002, 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.936 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.938 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:24.941 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.985484904000003, 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:02:24.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.985445214000002, 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.947 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:02:24.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.950202974, 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:02:24.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:02:24.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:388 3.16s0 3:6d55c691:::smithi18136759-16:c [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:24.956 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.551032125999999, 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.148899007, 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 388 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:24.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338147+0000", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.129e-06 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338155+0000", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1389999999999998e-06 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338160+0000", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338288+0000", 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128374 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338295+0000", 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7730000000000002e-06 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.342955+0000", 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046598580000000002 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343006+0000", 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1329000000000003e-05 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487043+0000", 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14403680199999999 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.16s0 60/53 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-17T16:02:24.961 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.545601417, 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14348618299999999, 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343575+0000", 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6400000000000003e-07 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343629+0000", 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4472000000000003e-05 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486547+0000", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.142917403 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486554+0000", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.503e-06 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486609+0000", 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4527000000000002e-05 2024-09-17T16:02:24.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487061+0000", 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045181399999999998 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "replica_scrub(pg: 3.15s1,from:0'0,to:56'8,epoch:60/53,start:3:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.402657477999998, 2024-09-17T16:02:24.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45217676400000001, 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486519+0000", 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486525+0000", 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5949999999999998e-06 2024-09-17T16:02:24.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938489+0000", 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45196369600000003 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938505+0000", 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6064000000000001e-05 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938656+0000", 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015132599999999999 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938666+0000", 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6320000000000003e-06 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938695+0000", 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9652e-05 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.969 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:447 3.1s0 3:8347559a:::smithi18136759-7:10 [read 97754~16934,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.548308595999998, 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014398077, 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 447 2024-09-17T16:02:24.970 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.971 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340869+0000", 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340875+0000", 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6630000000000004e-06 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340887+0000", 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.112e-05 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.972 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340896+0000", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344260+0000", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033644320000000001 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344265+0000", 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2479999999999998e-06 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.973 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353223+0000", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089583140000000002 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353351+0000", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128231 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355265+0000", 2024-09-17T16:02:24.974 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019143319999999999 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:448 3.1s0 3:8347559a:::smithi18136759-7:10 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.548252990000002, 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015642311999999998, 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.975 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 448 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.976 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340927+0000", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.569e-06 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340936+0000", 2024-09-17T16:02:24.977 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340942+0000", 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344342+0000", 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034007899999999999 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.978 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344344+0000", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.333e-06 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355309+0000", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010965557000000001 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355379+0000", 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9503000000000002e-05 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.979 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356565+0000", 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011865269999999999 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:449 3.1s0 3:8347559a:::smithi18136759-7:10 [read 41111~73577,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 20.548224046000001, 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015941349, 2024-09-17T16:02:24.980 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 449 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.981 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4400000000000001e-07 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340955+0000", 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0299999999999998e-06 2024-09-17T16:02:24.982 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340962+0000", 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.17e-06 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340968+0000", 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.983 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344372+0000", 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034039690000000002 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344373+0000", 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1400000000000001e-06 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355572+0000", 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011199565 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.984 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355632+0000", 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9210999999999999e-05 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356893+0000", 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012616330000000001 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.985 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:511 3.6s0 3:60a7d8b5:::smithi18136759-22:head [copy-from ver 12] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.514757485000001, 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016414522000000001, 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 511 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.986 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374421+0000", 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2189999999999998e-06 2024-09-17T16:02:24.987 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374429+0000", 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4929999999999998e-06 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374449+0000", 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0360000000000002e-05 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.988 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374459+0000", 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0559999999999994e-06 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374533+0000", 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4171999999999994e-05 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374651+0000", 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118495 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.989 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.383140+0000", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084888240000000007 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.384289+0000", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011493639999999999 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390716+0000", 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064265609999999999 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.990 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390833+0000", 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116978 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/53 ECSubWrite(tid=138, reqid=osd.14.0:138, at_version=74'39, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.485240184, 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.11895731, 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:24.991 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.992 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403952+0000", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5957e-05 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522019+0000", 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.118066429 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.993 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522040+0000", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1769000000000001e-05 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522199+0000", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015867999999999999 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522215+0000", 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5971999999999999e-05 2024-09-17T16:02:24.994 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522867+0000", 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065141500000000002 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522893+0000", 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6505000000000001e-05 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:24.995 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:24.996 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:24.997 INFO:tasks.rados.rados.0.smithi181.stdout:304: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-17T16:02:24.997 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 240258 2024-09-17T16:02:24.997 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 22474 2024-09-17T16:02:24.997 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 471893 2024-09-17T16:02:24.998 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:24.995+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:24.998 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:25.097 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:25.095+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:25.115 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.115 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:25.115 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:25.115 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:25.199 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:25.196+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:25.269 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.269 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:25.270 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:25.270 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:25.270 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.278 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:25.291 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 5 to smithi18136759-44 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: finishing write tid 6 to smithi18136759-44 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: oid 44 updating version 0 to 56 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: oid 44 version 56 is already newer than 55 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: oid 44 version 56 is already newer than 53 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: oid 44 version 56 is already newer than 52 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: oid 44 version 56 is already newer than 54 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 56 (ObjNum 124 snap 31 seq_num 124) dirty exists 2024-09-17T16:02:25.292 INFO:tasks.rados.rados.0.smithi181.stdout:305: left oid 44 (ObjNum 124 snap 31 seq_num 124) 2024-09-17T16:02:25.300 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:25.298+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 43/32 ECSubWriteReply(tid=24, last_complete=43'12, committed=1, applied=1))", 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:25.382 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 43.472157694000003, 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49241390899999998, 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907223+0000", 2024-09-17T16:02:25.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907224+0000", 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2299999999999999e-07 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.907228+0000", 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4689999999999999e-06 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399486+0000", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49225741699999997 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399495+0000", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.251e-06 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399587+0000", 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2224000000000003e-05 2024-09-17T16:02:25.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.399637+0000", 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9625e-05 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 43/32 ECSubWrite(tid=163, reqid=client.4560.0:215, at_version=43'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 43.470687783000002, 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65555535799999998, 2024-09-17T16:02:25.386 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908693+0000", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1500000000000003e-07 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:41.908987+0000", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029317500000000001 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562538+0000", 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.65355108900000003 2024-09-17T16:02:25.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562556+0000", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8773000000000001e-05 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562672+0000", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115352 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.562691+0000", 2024-09-17T16:02:25.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9102000000000001e-05 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564208+0000", 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0015170400000000001 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.564248+0000", 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0012000000000002e-05 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:264 3.es0 3:7408d595:::smithi18136759-48:head [append 0~704512] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.464335067999997, 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082026760000000008, 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 264 2024-09-17T16:02:25.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915045+0000", 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915047+0000", 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0650000000000001e-06 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915898+0000", 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085024099999999995 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915917+0000", 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9287999999999999e-05 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915926+0000", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0890000000000006e-06 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915992+0000", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5988999999999994e-05 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.915999+0000", 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5629999999999997e-06 2024-09-17T16:02:25.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918029+0000", 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002030652 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.918110+0000", 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0856e-05 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.919219+0000", 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0011090679999999999 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922879+0000", 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0036599639999999999 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923188+0000", 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00030846600000000002 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923248+0000", 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0435e-05 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:265 3.es0 3:7408d595:::smithi18136759-48:head [append 0~376832] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.462352481000003, 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013492013000000001, 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.397 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 265 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917028+0000", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917037+0000", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917413+0000", 2024-09-17T16:02:25.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00037550300000000001 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917425+0000", 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2345e-05 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917434+0000", 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8740000000000001e-06 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.400 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917491+0000", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6817000000000002e-05 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917495+0000", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1069999999999998e-06 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920406+0000", 2024-09-17T16:02:25.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029112610000000001 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.920476+0000", 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9451999999999998e-05 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921125+0000", 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00064978699999999998 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.923001+0000", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00187531 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930476+0000", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074754110000000004 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930520+0000", 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3838000000000003e-05 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.403 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:266 3.es0 3:7408d595:::smithi18136759-48:head [setxattr _header (56) in=7b] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.461926613000003, 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014603437, 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.404 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 266 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917454+0000", 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917455+0000", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5000000000000002e-07 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917459+0000", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.33e-06 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917466+0000", 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1570000000000003e-06 2024-09-17T16:02:25.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917472+0000", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1959999999999996e-06 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917515+0000", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3213000000000001e-05 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917521+0000", 2024-09-17T16:02:25.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5319999999999997e-06 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921833+0000", 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.004311942 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.921945+0000", 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112009 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922400+0000", 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00045485499999999999 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.925189+0000", 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0027889920000000001 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931982+0000", 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067935519999999996 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932057+0000", 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4908999999999997e-05 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:267 3.es0 3:7408d595:::smithi18136759-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.461898443000003, 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016824782, 2024-09-17T16:02:25.410 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 267 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917482+0000", 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917483+0000", 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917486+0000", 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.05e-06 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917493+0000", 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6190000000000002e-06 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917499+0000", 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2360000000000001e-06 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917545+0000", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5788000000000003e-05 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.917549+0000", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0149999999999997e-06 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922718+0000", 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051692420000000001 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.922863+0000", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014462599999999999 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932350+0000", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094864419999999994 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.932439+0000", 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8970999999999999e-05 2024-09-17T16:02:25.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.934307+0000", 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00186823 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.449097590000001, 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0094392299999999998, 2024-09-17T16:02:25.416 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930283+0000", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.42e-07 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930599+0000", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031603300000000001 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930714+0000", 2024-09-17T16:02:25.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000115344 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930722+0000", 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9640000000000003e-06 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930783+0000", 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1119999999999998e-05 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930789+0000", 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7180000000000001e-06 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939688+0000", 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088988690000000002 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939722+0000", 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4039999999999999e-05 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.420 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.448777962999998, 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0091445950000000002, 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:25.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930602+0000", 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930603+0000", 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5599999999999995e-07 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930688+0000", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4775000000000001e-05 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930749+0000", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1410999999999999e-05 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930756+0000", 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5420000000000002e-06 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931443+0000", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00068706500000000005 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931449+0000", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1369999999999999e-06 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939730+0000", 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082808859999999995 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939747+0000", 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7223000000000001e-05 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=179, reqid=client.4560.0:270, at_version=45'18, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 41.448686148999997, 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090693049999999997, 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.425 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930694+0000", 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930695+0000", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0900000000000002e-07 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930701+0000", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2840000000000003e-06 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930781+0000", 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0057999999999998e-05 2024-09-17T16:02:25.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.930788+0000", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5860000000000001e-06 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931794+0000", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001006215 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.931800+0000", 2024-09-17T16:02:25.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7660000000000004e-06 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939748+0000", 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079487499999999992 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.939764+0000", 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5237e-05 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.429 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=30, last_complete=46'18, committed=1, applied=1))", 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.459292847, 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24942447400000001, 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:25.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920088+0000", 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.72e-07 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.920091+0000", 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6620000000000001e-06 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168910+0000", 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.24881899499999999 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168924+0000", 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3968000000000001e-05 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.168997+0000", 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3919999999999997e-05 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.169512+0000", 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051455700000000001 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.433 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s2 46/32 ECSubWrite(tid=185, reqid=osd.11.0:185, at_version=46'29, trim_to=0'0, roll_forward_to=44'25))", 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.458081, 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080140109999999997, 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921299+0000", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921300+0000", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6499999999999999e-07 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921319+0000", 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8519e-05 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921341+0000", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2544999999999998e-05 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921346+0000", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9420000000000003e-06 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921389+0000", 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2620000000000002e-05 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.921394+0000", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3519999999999997e-06 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929283+0000", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007889007 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929313+0000", 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0460999999999999e-05 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.437 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 46/32 ECSubWrite(tid=37, reqid=osd.0.0:37, at_version=46'12, trim_to=0'0, roll_forward_to=44'10))", 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.454375906999999, 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010009911, 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.438 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925005+0000", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925020+0000", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4649e-05 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925042+0000", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2407000000000001e-05 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925050+0000", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3089999999999998e-06 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925097+0000", 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7763999999999999e-05 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925103+0000", 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0759999999999999e-06 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098665709999999993 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935014+0000", 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4459000000000002e-05 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s5 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.040382226999998, 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092966230000000004, 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.442 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338998+0000", 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338999+0000", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1199999999999999e-07 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339027+0000", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8099999999999999e-05 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339113+0000", 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5958000000000006e-05 2024-09-17T16:02:25.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339121+0000", 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5949999999999999e-06 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339202+0000", 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1150999999999996e-05 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339213+0000", 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0607e-05 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348258+0000", 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090445890000000004 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.348295+0000", 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7311000000000002e-05 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.446 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.0s1 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.037241841, 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0092835030000000002, 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342139+0000", 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342160+0000", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0803999999999999e-05 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342475+0000", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031531799999999998 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342484+0000", 2024-09-17T16:02:25.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1609999999999992e-06 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342552+0000", 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7761000000000004e-05 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.342562+0000", 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0316000000000001e-05 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351383+0000", 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088202019999999992 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351422+0000", 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9530000000000003e-05 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.13s4 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:25.451 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.033111945000002, 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081316900000000004, 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346268+0000", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346269+0000", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346318+0000", 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8843e-05 2024-09-17T16:02:25.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346402+0000", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4646000000000002e-05 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346411+0000", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3890000000000005e-06 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346518+0000", 2024-09-17T16:02:25.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000107693 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346525+0000", 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5200000000000003e-06 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354365+0000", 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078396709999999994 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.455 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354400+0000", 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5627999999999998e-05 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs1 64/32 ECSubWrite(tid=68, reqid=osd.0.0:68, at_version=64'32, trim_to=0'0, roll_forward_to=62'30))", 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.033011173999999, 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0081747250000000007, 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.456 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346369+0000", 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346370+0000", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6e-07 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346392+0000", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2419e-05 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346501+0000", 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010902 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346509+0000", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.0609999999999996e-06 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346569+0000", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0405999999999997e-05 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346579+0000", 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4700000000000008e-06 2024-09-17T16:02:25.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354505+0000", 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079256489999999999 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354544+0000", 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9440000000000002e-05 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.460 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.es0 68/32 ECSubWriteReply(tid=68, last_complete=68'49, committed=1, applied=1))", 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 18.985650694, 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44479439100000001, 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:25.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393730+0000", 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7499999999999999e-07 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.393734+0000", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9659999999999998e-06 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837925+0000", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.44419083199999998 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.837936+0000", 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0349e-05 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838002+0000", 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6196000000000001e-05 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.838524+0000", 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052227299999999996 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.464 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:634 3.es0 3:7b038293:::smithi18136759-50:head [stat,delete] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9506802829999996, 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010543408000000001, 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 634 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.465 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428700+0000", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428702+0000", 2024-09-17T16:02:25.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9709999999999998e-06 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428710+0000", 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2339999999999994e-06 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428728+0000", 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7623000000000001e-05 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428736+0000", 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2309999999999996e-06 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428803+0000", 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7249000000000006e-05 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.428809+0000", 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7819999999999999e-06 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.431048+0000", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022392660000000002 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.431135+0000", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6567e-05 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.432031+0000", 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00089615400000000003 2024-09-17T16:02:25.469 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.434430+0000", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0023985130000000001 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439144+0000", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0047146560000000002 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439244+0000", 2024-09-17T16:02:25.470 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9161999999999997e-05 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:639 3.es0 3:7408d595:::smithi18136759-48:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 7.9397266069999999, 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080763160000000001, 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:25.471 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 639 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.472 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439654+0000", 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439656+0000", 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9199999999999998e-06 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.473 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439660+0000", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6179999999999999e-06 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.439663+0000", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8210000000000002e-06 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.442607+0000", 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0029446619999999998 2024-09-17T16:02:25.474 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.442716+0000", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000109127 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.443266+0000", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00054992799999999998 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.445211+0000", 2024-09-17T16:02:25.475 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00194429 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.447666+0000", 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002455704 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:17.447730+0000", 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3910000000000003e-05 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.476 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.3s4 81/53 ECSubWrite(tid=74, reqid=osd.15.0:74, at_version=81'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.427345+0000", 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 5.9520356300000001, 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.018580622000000001, 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.427345+0000", 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.477 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.427345+0000", 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.427345+0000", 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2600000000000001e-07 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.478 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.427356+0000", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1077000000000001e-05 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442150+0000", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014793426 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442157+0000", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8779999999999998e-06 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.479 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442231+0000", 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4073999999999996e-05 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442241+0000", 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6760000000000002e-06 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445905+0000", 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003664095 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.480 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445925+0000", 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0169999999999998e-05 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.481 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.482 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:25.483 INFO:tasks.rados.rados.0.smithi181.stdout:304: done (2 left) 2024-09-17T16:02:25.483 INFO:tasks.rados.rados.0.smithi181.stdout:305: done (1 left) 2024-09-17T16:02:25.483 INFO:tasks.rados.rados.0.smithi181.stdout:307: done (0 left) 2024-09-17T16:02:25.483 INFO:tasks.rados.rados.0.smithi181.stdout:308: copy_from oid 16 from oid 38 current snap is 32 2024-09-17T16:02:25.483 INFO:tasks.rados.rados.0.smithi181.stdout:309: snap_create 2024-09-17T16:02:25.484 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:25.484 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:25.480+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:25.486 INFO:tasks.rados.rados.0.smithi181.stdout:308: finishing copy_from racing read to smithi18136759-16 2024-09-17T16:02:25.488 INFO:tasks.rados.rados.0.smithi181.stdout:308: finishing copy_from to smithi18136759-16 2024-09-17T16:02:25.488 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 39 (ObjNum 161 snap 0 seq_num 1) dirty exists 2024-09-17T16:02:25.583 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:25.581+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:25.628 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.629 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:25.629 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:25.629 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.637 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:25.684 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:25.681+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:25.740 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.740 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:25.740 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:25.741 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:25.741 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.749 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:25.785 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:25.782+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:25.852 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=151, reqid=client.4560.0:182, at_version=38'14, trim_to=0'0, roll_forward_to=38'11))", 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 48.963723608999999, 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052702679999999998, 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.853 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886294+0000", 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886295+0000", 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4200000000000004e-07 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886312+0000", 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6361000000000001e-05 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886343+0000", 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1229000000000002e-05 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886351+0000", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4179999999999997e-06 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886421+0000", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0017000000000004e-05 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.886431+0000", 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.363e-06 2024-09-17T16:02:25.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891526+0000", 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050953580000000004 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:36.891565+0000", 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.858e-05 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.925918733000003, 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0054480279999999997, 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924099+0000", 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.9500000000000001e-07 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924122+0000", 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2844999999999999e-05 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924159+0000", 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6534e-05 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924166+0000", 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.742e-06 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924210+0000", 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4641999999999997e-05 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924220+0000", 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7580000000000004e-06 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929512+0000", 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.005292265 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929547+0000", 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5046999999999998e-05 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=51, reqid=osd.12.0:51, at_version=46'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:25.863 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.925811594999999, 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053997719999999997, 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924206+0000", 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924207+0000", 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.865 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924223+0000", 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6835000000000001e-05 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924252+0000", 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8285000000000001e-05 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924259+0000", 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.8630000000000002e-06 2024-09-17T16:02:25.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924320+0000", 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1709000000000002e-05 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.924329+0000", 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8799999999999997e-06 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929582+0000", 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052531169999999999 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929606+0000", 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3913e-05 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.924989355000001, 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:02:25.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:02:25.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:02:25.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.875 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:25.876 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:25.876 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.922630392999999, 2024-09-17T16:02:25.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:02:25.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 46/32 ECSubWrite(tid=187, reqid=osd.11.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=46'29))", 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.920229751000001, 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0052541769999999996, 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.882 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929788+0000", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929789+0000", 2024-09-17T16:02:25.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7999999999999995e-07 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929800+0000", 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1341e-05 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929829+0000", 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8642000000000001e-05 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929837+0000", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929901+0000", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4067999999999997e-05 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.929907+0000", 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.609e-06 2024-09-17T16:02:25.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934990+0000", 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050827579999999997 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935042+0000", 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2676999999999997e-05 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.886 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.504280671, 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060163380000000004, 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.887 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345737+0000", 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8900000000000001e-07 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.888 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345763+0000", 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5772e-05 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345830+0000", 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6855999999999998e-05 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345839+0000", 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.0270000000000001e-06 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.889 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345913+0000", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3454999999999994e-05 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345924+0000", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351719+0000", 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057956309999999999 2024-09-17T16:02:25.890 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.351754+0000", 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4439e-05 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.504251840999999, 2024-09-17T16:02:25.891 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087141639999999999, 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:25.892 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345787+0000", 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0109e-05 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.893 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346312+0000", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052588999999999999 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346324+0000", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1445e-05 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346394+0000", 2024-09-17T16:02:25.894 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9604000000000006e-05 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346403+0000", 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.611e-06 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354448+0000", 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080448199999999994 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.895 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354480+0000", 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2329000000000002e-05 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.503819502999999, 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082305039999999996, 2024-09-17T16:02:25.896 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.897 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346199+0000", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7300000000000002e-07 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346223+0000", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4442000000000001e-05 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346360+0000", 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136792 2024-09-17T16:02:25.898 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346374+0000", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3669000000000001e-05 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346429+0000", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4898999999999997e-05 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346437+0000", 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:02:25.899 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354393+0000", 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079559079999999994 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354429+0000", 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6168999999999999e-05 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.900 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.10s1 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 23.503780502000001, 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083156649999999999, 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:25.901 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346238+0000", 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.902 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346256+0000", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8805999999999999e-05 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346408+0000", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000151519 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346417+0000", 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6619999999999994e-06 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.903 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346513+0000", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6127999999999998e-05 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346523+0000", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9750000000000002e-06 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354521+0000", 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079977780000000005 2024-09-17T16:02:25.904 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354553+0000", 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2614000000000003e-05 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 81/53 ECSubWrite(tid=277, reqid=osd.11.0:277, at_version=81'62, trim_to=0'0, roll_forward_to=79'60))", 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4187682329999998, 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010651089000000001, 2024-09-17T16:02:25.905 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.906 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4499999999999998e-07 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431286+0000", 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6235000000000001e-05 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.907 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431365+0000", 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8455999999999996e-05 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431373+0000", 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5860000000000005e-06 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431445+0000", 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1258000000000001e-05 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.908 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431455+0000", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.007e-05 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441861+0000", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010406072000000001 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441901+0000", 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0166999999999997e-05 2024-09-17T16:02:25.909 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 81/53 ECSubWrite(tid=278, reqid=osd.11.0:278, at_version=81'50, trim_to=0'0, roll_forward_to=77'48))", 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4187265509999998, 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010640938000000001, 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.910 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431292+0000", 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:02:25.911 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431315+0000", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2889000000000001e-05 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431418+0000", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103299 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431425+0000", 2024-09-17T16:02:25.912 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9430000000000003e-06 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431465+0000", 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0303999999999999e-05 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431473+0000", 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2770000000000005e-06 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.913 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441909+0000", 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01043585 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441932+0000", 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3074e-05 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 81/32 ECSubWrite(tid=100, reqid=osd.0.0:100, at_version=81'47, trim_to=0'0, roll_forward_to=79'45))", 2024-09-17T16:02:25.914 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4161431740000001, 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010862208999999999, 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.915 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.48e-07 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433889+0000", 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3553000000000001e-05 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.916 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433919+0000", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0408000000000001e-05 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433925+0000", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5890000000000002e-06 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433972+0000", 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7898000000000001e-05 2024-09-17T16:02:25.917 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433983+0000", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0239e-05 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444701+0000", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010717925999999999 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444737+0000", 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6448000000000001e-05 2024-09-17T16:02:25.918 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 81/32 ECSubWrite(tid=101, reqid=osd.0.0:101, at_version=81'49, trim_to=0'0, roll_forward_to=79'45))", 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.414142182, 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098407489999999993, 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.919 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:25.920 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435897+0000", 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0579999999999999e-05 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435930+0000", 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2873000000000002e-05 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.921 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435938+0000", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5709999999999998e-06 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435990+0000", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1897000000000001e-05 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435998+0000", 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6520000000000006e-06 2024-09-17T16:02:25.922 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445684+0000", 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096856419999999995 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445717+0000", 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2938000000000002e-05 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.923 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 81/53 ECSubWrite(tid=45, reqid=osd.8.0:45, at_version=81'22, trim_to=0'0, roll_forward_to=77'20))", 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4132592239999999, 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090338850000000002, 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.924 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1099999999999995e-07 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.925 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436775+0000", 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5614e-05 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436802+0000", 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6704999999999999e-05 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436818+0000", 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5858e-05 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.926 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436875+0000", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7402999999999999e-05 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436885+0000", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7860000000000008e-06 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445759+0000", 2024-09-17T16:02:25.927 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088739249999999995 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445793+0000", 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3983000000000003e-05 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 81/53 ECSubWrite(tid=46, reqid=osd.8.0:46, at_version=81'22, trim_to=0'0, roll_forward_to=77'20))", 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.412971711, 2024-09-17T16:02:25.928 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087742510000000003, 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:25.929 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.74e-07 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437060+0000", 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3212e-05 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.930 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437074+0000", 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4769999999999999e-05 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437082+0000", 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437128+0000", 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6455000000000002e-05 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.931 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437136+0000", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1740000000000003e-06 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445796+0000", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086608529999999996 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445821+0000", 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4159e-05 2024-09-17T16:02:25.932 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=101, reqid=osd.2.0:100, at_version=0'0, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4125152090000004, 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078549550000000003, 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.933 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:25.934 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437511+0000", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437539+0000", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7525000000000002e-05 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437545+0000", 2024-09-17T16:02:25.935 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8100000000000003e-06 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437582+0000", 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7070000000000003e-05 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437587+0000", 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3070000000000002e-06 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.936 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445327+0000", 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077395270000000004 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445358+0000", 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1210000000000001e-05 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=102, reqid=osd.2.0:102, at_version=81'87, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:25.937 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4113588759999995, 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080538369999999995, 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.938 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438660+0000", 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.540000000000001e-07 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438689+0000", 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8722e-05 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.939 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438734+0000", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5291999999999999e-05 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438742+0000", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1729999999999995e-06 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438789+0000", 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6764999999999997e-05 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.940 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438798+0000", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446667+0000", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078683329999999999 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446713+0000", 2024-09-17T16:02:25.941 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6289999999999999e-05 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=103, reqid=osd.2.0:103, at_version=81'89, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4103131810000002, 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070349870000000004, 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.942 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439706+0000", 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:02:25.943 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439725+0000", 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8916999999999999e-05 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439760+0000", 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5351000000000001e-05 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.944 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439782+0000", 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1793e-05 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442292+0000", 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025106099999999999 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442302+0000", 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8660000000000001e-06 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.945 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446717+0000", 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044147750000000001 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446740+0000", 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2884999999999999e-05 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.946 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=104, reqid=osd.2.0:103, at_version=0'0, trim_to=0'0, roll_forward_to=81'89))", 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.448000+0000", 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.4020183370000003, 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0059610419999999997, 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448000+0000", 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.947 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448000+0000", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448000+0000", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7299999999999995e-07 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448010+0000", 2024-09-17T16:02:25.948 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2560000000000008e-06 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448037+0000", 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7438999999999999e-05 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448044+0000", 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2180000000000002e-06 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.949 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448102+0000", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7469000000000002e-05 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448111+0000", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4050000000000006e-06 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453916+0000", 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058049419999999996 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:25.950 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453961+0000", 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.464e-05 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:25.951 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:25.953 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:25.953 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:25.950+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:26.054 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:26.052+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:26.058 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:26.058 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:26.059 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:26.059 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:26.067 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:26.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:26.155 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:26.153+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:26.210 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:26.210 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:26.210 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:26.210 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:26.210 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:26.218 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:26.260 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:26.258+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:26.277 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:26.278 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:26.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:26.301 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:26.322 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:26.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:26.344 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:26.344 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:26.358 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:26.356+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.506337604999999, 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:02:26.364 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:02:26.368 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.369 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.505436731000003, 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:02:26.372 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:02:26.376 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.501346904000002, 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:02:26.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.381 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.498939943000003, 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:02:26.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.497114564999997, 2024-09-17T16:02:26.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:26.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:02:26.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.496966489999998, 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:26.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:02:26.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.496316606000001, 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:02:26.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.493739243, 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.400 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.401 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.402 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.403 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.404 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 52.493248070999996, 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.405 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.406 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:02:26.407 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.408 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 52/32 ECSubWrite(tid=89, reqid=client.4560.0:340, at_version=52'11, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 35.988326661000002, 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15958299200000001, 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.409 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372994+0000", 2024-09-17T16:02:26.410 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373011+0000", 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6280999999999999e-05 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531935+0000", 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15892432300000001 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.411 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531949+0000", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532026+0000", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6526000000000006e-05 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532033+0000", 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:02:26.412 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532538+0000", 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050494099999999996 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532576+0000", 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8059999999999998e-05 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.413 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 24.022802191, 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058173629999999999, 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:26.414 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5199999999999997e-07 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.415 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338580+0000", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1075999999999996e-05 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338633+0000", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3263999999999997e-05 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338646+0000", 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3326000000000001e-05 2024-09-17T16:02:26.416 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339218+0000", 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057138000000000002 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339229+0000", 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1487e-05 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.417 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344292+0000", 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050634460000000001 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344335+0000", 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2831999999999999e-05 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:602 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~606208] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:26.418 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.929702502, 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085762449999999997, 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 602 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:26.419 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431620+0000", 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5469999999999999e-06 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.420 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432327+0000", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000707079 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432340+0000", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3021e-05 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432358+0000", 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8019000000000001e-05 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.421 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432480+0000", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121995 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432491+0000", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0457e-05 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434624+0000", 2024-09-17T16:02:26.422 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021328250000000001 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434764+0000", 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014050999999999999 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.436049+0000", 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012844969999999999 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.423 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439929+0000", 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038802730000000001 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440073+0000", 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014401099999999999 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440194+0000", 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121011 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.424 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:603 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~688128] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.928892192999999, 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089592730000000002, 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:26.425 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 603 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.426 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432429+0000", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432988+0000", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055916400000000002 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432997+0000", 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8310000000000007e-06 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.427 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433001+0000", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433056+0000", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4994999999999998e-05 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433061+0000", 2024-09-17T16:02:26.428 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437100+0000", 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040393169999999997 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437171+0000", 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0548000000000003e-05 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.429 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437907+0000", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073646200000000001 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439930+0000", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002022565 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441283+0000", 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013528189999999999 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.430 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441387+0000", 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010435 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:604 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~475136] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.928304872, 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086384329999999992, 2024-09-17T16:02:26.431 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 604 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.432 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433016+0000", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6899999999999997e-07 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433437+0000", 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042084799999999998 2024-09-17T16:02:26.433 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433445+0000", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.354e-06 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433450+0000", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1460000000000004e-06 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433526+0000", 2024-09-17T16:02:26.434 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5370000000000005e-05 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.22e-06 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438550+0000", 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050189079999999999 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:26.435 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438619+0000", 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9138000000000001e-05 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439350+0000", 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073165800000000005 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441219+0000", 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001868747 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.436 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441548+0000", 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032912700000000002 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441654+0000", 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010534799999999999 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.437 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:605 3.11s0 3:89c7af4b:::smithi18136759-34:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.927859091, 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086379049999999995, 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 605 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.438 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:26.439 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433484+0000", 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2540000000000001e-05 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433488+0000", 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1790000000000001e-06 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.440 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433491+0000", 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.915e-06 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433560+0000", 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9307000000000006e-05 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433565+0000", 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8049999999999997e-06 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.441 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440435+0000", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068700439999999996 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440548+0000", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112904 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440860+0000", 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031159299999999998 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.442 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441710+0000", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085002099999999996 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441963+0000", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025321099999999999 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442099+0000", 2024-09-17T16:02:26.443 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013592399999999999 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:606 3.11s0 3:89c7af4b:::smithi18136759-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 11.927801015, 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010718325000000001, 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:26.444 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 606 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:26.445 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6800000000000001e-07 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433522+0000", 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.052e-06 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.446 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9580000000000002e-06 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433536+0000", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4750000000000004e-06 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433588+0000", 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2129999999999997e-05 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.447 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433592+0000", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9090000000000002e-06 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441080+0000", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007488476 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441139+0000", 2024-09-17T16:02:26.448 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8468000000000003e-05 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442343+0000", 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012042520000000001 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442403+0000", 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9846999999999998e-05 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.449 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.444237+0000", 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00183439 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=148, reqid=osd.14.0:148, at_version=81'37, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.9231795040000002, 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065669389999999999, 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.450 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.451 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9900000000000002e-07 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438158+0000", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7248000000000002e-05 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438229+0000", 2024-09-17T16:02:26.452 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1317999999999997e-05 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438248+0000", 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.827e-05 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438320+0000", 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2342999999999995e-05 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.453 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438331+0000", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444669+0000", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063379129999999997 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444708+0000", 2024-09-17T16:02:26.454 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8578999999999998e-05 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=150, reqid=osd.14.0:150, at_version=81'39, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.9219614099999998, 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057892539999999998, 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.455 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.456 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6700000000000003e-07 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439380+0000", 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0604000000000001e-05 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439418+0000", 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7707000000000003e-05 2024-09-17T16:02:26.457 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439426+0000", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3289999999999994e-06 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439492+0000", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5801000000000002e-05 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439505+0000", 2024-09-17T16:02:26.458 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2699e-05 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445102+0000", 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055975479999999999 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445148+0000", 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5899000000000002e-05 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.459 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.9157161760000001, 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082580629999999995, 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:26.460 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445605+0000", 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029e-06 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.461 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445725+0000", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120361 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445763+0000", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7991000000000003e-05 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445773+0000", 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0039e-05 2024-09-17T16:02:26.462 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445830+0000", 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6909e-05 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445841+0000", 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0327e-05 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.463 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453823+0000", 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079825799999999995 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453862+0000", 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8826999999999999e-05 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.464 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 6.9126794560000002, 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055985469999999997, 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.465 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448735+0000", 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3511000000000006e-05 2024-09-17T16:02:26.466 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448773+0000", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8169e-05 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448782+0000", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7509999999999997e-06 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448846+0000", 2024-09-17T16:02:26.467 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4108000000000004e-05 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448877+0000", 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1097999999999999e-05 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454215+0000", 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053383129999999999 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:26.468 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454239+0000", 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3830000000000001e-05 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:26.469 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:26.470 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:26.471 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:26.468+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:26.471 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:26.471 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:26.472 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:26.472 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:26.473 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:26.473 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:26.473 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:26.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:26.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:26.474 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:26.475 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:26.475 INFO:tasks.rados.rados.0.smithi181.stdout:308: done (1 left) 2024-09-17T16:02:26.476 INFO:tasks.rados.rados.0.smithi181.stdout:309: done (0 left) 2024-09-17T16:02:26.476 INFO:tasks.rados.rados.0.smithi181.stdout:310: delete oid 30 current snap is 33 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:310: done (0 left) 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:311: rmattr oid 30 current snap is 33 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:311: done (0 left) 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:312: read oid 9 snap -1 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:312: expect (ObjNum 118 snap 29 seq_num 118) 2024-09-17T16:02:26.480 INFO:tasks.rados.rados.0.smithi181.stdout:313: snap_create 2024-09-17T16:02:26.488 INFO:tasks.rados.rados.0.smithi181.stdout:312: expect (ObjNum 118 snap 29 seq_num 118) 2024-09-17T16:02:26.488 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:26.510 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 785600 2024-09-17T16:02:26.529 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1555103 2024-09-17T16:02:26.571 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:26.569+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:26.578 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:26.578 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:26.578 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:26.578 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:26.587 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:26.618 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:26.618 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:26.638 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:26.660 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:26.660 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:26.672 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:26.670+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:26.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:26.679+0000 7fb275d6e640 0 -- 172.21.15.6:0/3381477916 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fb270221b10 msgr2=0x7fb270223f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T16:02:26.730 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:26.731 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:26.739 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:26.774 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:26.772+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:26.843 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:26.843 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:26.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:26.895 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:26.893+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:26.906 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:26.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:26.968 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:26.968 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:26.975 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:26.973+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:26.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:27.075 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:27.073+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:27.121 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.122 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:27.176 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:27.174+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:27.233 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.233 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:27.277 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:27.275+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:27.345 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.345 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:27.378 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:27.377+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:27.449 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:27.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:27.449 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-17T16:02:27.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-17T16:02:27.457 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.457 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:27.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:02:27.486 INFO:tasks.rados.rados.0.smithi181.stdout:312: done (1 left) 2024-09-17T16:02:27.486 INFO:tasks.rados.rados.0.smithi181.stdout:313: done (0 left) 2024-09-17T16:02:27.486 INFO:tasks.rados.rados.0.smithi181.stdout:314: read oid 28 snap -1 2024-09-17T16:02:27.486 INFO:tasks.rados.rados.0.smithi181.stdout:314: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-17T16:02:27.486 INFO:tasks.rados.rados.0.smithi181.stdout:315: delete oid 21 current snap is 34 2024-09-17T16:02:27.486 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:27.483+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:27.490 INFO:tasks.rados.rados.0.smithi181.stdout:314: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-17T16:02:27.494 INFO:tasks.rados.rados.0.smithi181.stdout:314: done (1 left) 2024-09-17T16:02:27.494 INFO:tasks.rados.rados.0.smithi181.stdout:315: done (0 left) 2024-09-17T16:02:27.494 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 34 2024-09-17T16:02:27.494 INFO:tasks.rados.rados.0.smithi181.stdout:316: seq_num 128 ranges {1916928=606208,2523136=507904,3031040=573440,3604480=475136} 2024-09-17T16:02:27.507 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18136759-50 from 1916928 to 2523136 tid 1 2024-09-17T16:02:27.512 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18136759-50 from 2523136 to 3031040 tid 2 2024-09-17T16:02:27.513 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18136759-50 from 3031040 to 3604480 tid 3 2024-09-17T16:02:27.517 INFO:tasks.rados.rados.0.smithi181.stdout:316: writing smithi18136759-50 from 3604480 to 4079616 tid 4 2024-09-17T16:02:27.517 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 49 current snap is 34 2024-09-17T16:02:27.517 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 1 to smithi18136759-50 2024-09-17T16:02:27.517 INFO:tasks.rados.rados.0.smithi181.stdout:317: seq_num 129 ranges {32768=622592,655360=655360,1310720=688128,1998848=212992} 2024-09-17T16:02:27.520 INFO:tasks.rados.rados.0.smithi181.stdout:317: writing smithi18136759-49 from 32768 to 655360 tid 1 2024-09-17T16:02:27.524 INFO:tasks.rados.rados.0.smithi181.stdout:317: writing smithi18136759-49 from 655360 to 1310720 tid 2 2024-09-17T16:02:27.528 INFO:tasks.rados.rados.0.smithi181.stdout:317: writing smithi18136759-49 from 1310720 to 1998848 tid 3 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:317: writing smithi18136759-49 from 1998848 to 2211840 tid 4 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:318: snap_create 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 2 to smithi18136759-50 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 3 to smithi18136759-50 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 4 to smithi18136759-50 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 5 to smithi18136759-50 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: finishing write tid 6 to smithi18136759-50 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 50 updating version 0 to 110 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 50 version 110 is already newer than 109 2024-09-17T16:02:27.530 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 50 version 110 is already newer than 107 2024-09-17T16:02:27.531 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 50 version 110 is already newer than 108 2024-09-17T16:02:27.531 INFO:tasks.rados.rados.0.smithi181.stdout:316: oid 50 updating version 110 to 111 2024-09-17T16:02:27.531 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 111 (ObjNum 128 snap 34 seq_num 128) dirty exists 2024-09-17T16:02:27.531 INFO:tasks.rados.rados.0.smithi181.stdout:316: left oid 50 (ObjNum 128 snap 34 seq_num 128) 2024-09-17T16:02:27.531 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 1 to smithi18136759-49 2024-09-17T16:02:27.532 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 2 to smithi18136759-49 2024-09-17T16:02:27.534 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 3 to smithi18136759-49 2024-09-17T16:02:27.534 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 4 to smithi18136759-49 2024-09-17T16:02:27.574 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.575 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.580 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:27.577+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:27.583 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:27.681 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:27.678+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:27.725 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.725 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 89, 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "4345ca04-2628-4bef-a537-6837268b96c9", 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-09-17T16:01:02.569356+0000", 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-09-17T16:02:27.474703+0000", 2024-09-17T16:02:27.782 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-09-17T16:02:03.333624+0000", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-09-17T16:01:06.794253+0000", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5799936, 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 7, 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-17T16:02:27.783 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 3, 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 16, 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "luminous", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:14.963100+0000", 2024-09-17T16:02:27.784 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 2, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 1, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:02:27.785 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 1, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 1, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 1, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 1, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 1, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:02:27.786 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "25", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:02:27.787 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:02:27.788 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:02:27.789 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-09-17T16:02:27.790 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 15.380000114440918, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 15.380000114440918, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.12999999523162842, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-09-17T16:02:27.791 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "rbd", 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:17.236630+0000", 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 2, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 1, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:02:27.792 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 8, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 8, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 8, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 8, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 8, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:02:27.793 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "28", 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 28, 2024-09-17T16:02:27.794 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:02:27.795 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:02:27.796 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:02:27.797 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 4.0199999809265137, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 4.0199999809265137, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.56000000238418579, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2.25, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2.25, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_0", 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-09-17T16:01:29.829066+0000", 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 3, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 7, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 5, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 1, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-17T16:02:27.799 INFO:teuthology.orchestra.run.smithi006.stdout: "is_stretch_pool": false, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 26, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 19, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 26, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 26, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 26, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-09-17T16:02:27.800 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "89", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "53", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 35, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 89, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-09-17T16:02:27.801 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "shecprofile", 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-09-17T16:02:27.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 16384, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-09-17T16:02:27.803 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "rados": {} 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "282c4269-7e53-43fc-af4b-3b133d82837c", 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.804 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 865094279 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/865094279", 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6811/865094279", 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6815/865094279", 2024-09-17T16:02:27.809 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6813/865094279", 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "66e20d10-2bb0-4130-93a5-45bcdd2de605", 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.810 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 21, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6801", 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6803", 2024-09-17T16:02:27.811 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6805", 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6807", 2024-09-17T16:02:27.812 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6813", 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6815", 2024-09-17T16:02:27.813 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6809", 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6811", 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1460756480 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.814 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6803/1460756480", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6807/1460756480", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6815/1460756480", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6811/1460756480", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "c6ec9e20-133a-471b-b63c-6453a7bde86c", 2024-09-17T16:02:27.815 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 32, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.816 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6816", 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6817", 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.817 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6818", 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6819", 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.818 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6822", 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6823", 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.819 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6820", 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6821", 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1905413450 2024-09-17T16:02:27.820 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6817/1905413450", 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6819/1905413450", 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6823/1905413450", 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6821/1905413450", 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.821 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3491a731-ca96-4981-be73-e005e00c315f", 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6810", 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6811", 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.823 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6812", 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6813", 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6816", 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6817", 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6814", 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6815", 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2918557179 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.826 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6811/2918557179", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6813/2918557179", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6817/2918557179", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6815/2918557179", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3bdfaa34-bc10-4cec-b5c2-86e05d444ffd", 2024-09-17T16:02:27.827 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.828 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6800", 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.829 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.830 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.831 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2257685015 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/2257685015", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6803/2257685015", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6807/2257685015", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6805/2257685015", 2024-09-17T16:02:27.832 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2", 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.833 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 0, 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6824", 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6825", 2024-09-17T16:02:27.834 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6826", 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6827", 2024-09-17T16:02:27.835 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6830", 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6831", 2024-09-17T16:02:27.836 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6828", 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6829", 2024-09-17T16:02:27.837 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2399951648 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6825/2399951648", 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6827/2399951648", 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6831/2399951648", 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6829/2399951648", 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.838 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "ddd3edea-260a-4ec9-8533-841ed933fdb8", 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6824", 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6825", 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.840 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6826", 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6827", 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.841 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6830", 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6831", 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.842 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6828", 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6829", 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1225429423 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6825/1225429423", 2024-09-17T16:02:27.843 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6827/1225429423", 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6831/1225429423", 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6829/1225429423", 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7afc1a18-98c2-4a58-a09d-1b4f474d4dc4", 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 0, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.844 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 57, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 65, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6818", 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6819", 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6820", 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6821", 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6824", 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6825", 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6822", 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6823", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3415295348 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6819/3415295348", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6821/3415295348", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6825/3415295348", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6823/3415295348", 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: "exists" 2024-09-17T16:02:27.849 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 8, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "3e45f80f-856b-4d3f-b002-255f16a14f6b", 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.850 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6824", 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6825", 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6826", 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6827", 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.852 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6830", 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6831", 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.853 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6828", 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6829", 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2216439459 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.854 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6825/2216439459", 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6827/2216439459", 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6831/2216439459", 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6829/2216439459", 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 9, 2024-09-17T16:02:27.855 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b415c423-24de-4d86-bec4-ea330364ff96", 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 25, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6816", 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6817", 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6818", 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6819", 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6822", 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6823", 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6820", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6821", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1727176716 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6817/1727176716", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6819/1727176716", 2024-09-17T16:02:27.860 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6823/1727176716", 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6821/1727176716", 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 10, 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "e471e1cb-9c0b-4c99-8622-59eb7956770d", 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.861 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6808", 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6809", 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.863 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6810", 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6811", 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.864 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6814", 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6815", 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.865 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6812", 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6813", 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3604183526 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.866 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6809/3604183526", 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6811/3604183526", 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6815/3604183526", 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6813/3604183526", 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 11, 2024-09-17T16:02:27.867 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "c2bf4255-7ce1-4673-bedf-fd055d841fac", 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6826", 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6827", 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6828", 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6829", 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6832", 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6833", 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.871 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6830", 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6831", 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3420013159 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.872 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6827/3420013159", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6829/3420013159", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6833/3420013159", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6831/3420013159", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 12, 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "4c16f3ee-f427-41d5-b0ba-9e676ddbe032", 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.873 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 23, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-09-17T16:02:27.874 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6817", 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6818", 2024-09-17T16:02:27.875 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6819", 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.876 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6822", 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6823", 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6820", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6821", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2310487797 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6817/2310487797", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6819/2310487797", 2024-09-17T16:02:27.878 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6823/2310487797", 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6821/2310487797", 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 13, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "0a4386aa-1a59-44c8-8d1e-7d43bf10de2e", 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.879 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6800", 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6802", 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6804", 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6806", 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6812", 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6814", 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6808", 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.883 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.50:6810", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1210437211 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.50:6802/1210437211", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.50:6806/1210437211", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.50:6814/1210437211", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.50:6810/1210437211", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 14, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "4c54e35c-e226-4d8c-bad0-51dd1cd95fe0", 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 53, 2024-09-17T16:02:27.885 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6800", 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6801", 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6802", 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6803", 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.887 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6806", 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6807", 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.888 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6804", 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.169:6805", 2024-09-17T16:02:27.889 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 4170280864 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.169:6801/4170280864", 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.169:6803/4170280864", 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.169:6807/4170280864", 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.169:6805/4170280864", 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.890 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 15, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "0281e750-f74a-4760-bbef-fbfa4127d06b", 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 22, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 65, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.891 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6802", 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6803", 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6804", 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6805", 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.893 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6808", 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6809", 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-09-17T16:02:27.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6806", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v1", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.181:6807", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2815528414 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.181:6803/2815528414", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.181:6805/2815528414", 2024-09-17T16:02:27.895 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.181:6809/2815528414", 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.181:6807/2815528414", 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.896 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.014883+0000", 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.897 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.677746+0000", 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.838957+0000", 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.321698+0000", 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.899 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:02:25.617972+0000", 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.900 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:02:25.371442+0000", 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.517939+0000", 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.664321+0000", 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 65 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 8, 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.902 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:02:26.141597+0000", 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 9, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.903 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:02:25.015167+0000", 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 10, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.124128+0000", 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 11, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.554475+0000", 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 12, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.905 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:13.868839+0000", 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 13, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.912548+0000", 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 14, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:01:12.882601+0000", 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 15, 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540719139924082687, 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-09-17T16:02:26.664633+0000", 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [ 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "2.7", 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-09-17T16:02:27.908 INFO:teuthology.orchestra.run.smithi006.stdout: 13, 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: 15 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [], 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [], 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": {}, 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-09-17T16:02:27.909 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "shecprofile": { 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "c": "2", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-device-class": "", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-num-failure-domains": "0", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-osds-per-failure-domain": "0", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-root": "default", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "4", 2024-09-17T16:02:27.910 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "3", 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "shecprofile", 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "shec", 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "multiple" 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [ 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "snaps": [ 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 2, 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 1 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.911 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 8, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 3 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 13, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 3 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 20, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 2 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 23, 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 1 2024-09-17T16:02:27.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "begin": 26, 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "length": 2 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-09-17T16:02:27.913 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-09-17T16:02:27.914 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-09-17T16:02:27.914 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-09-17T16:02:27.914 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-09-17T16:02:27.914 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:27.914 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:27.915 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:27.780+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:27.916 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:27.916 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:27.918 INFO:tasks.thrashosds.thrasher:j is {'epoch': 89, 'fsid': '4345ca04-2628-4bef-a537-6837268b96c9', 'created': '2024-09-17T16:01:02.569356+0000', 'modified': '2024-09-17T16:02:27.474703+0000', 'last_up_change': '2024-09-17T16:02:03.333624+0000', 'last_in_change': '2024-09-17T16:01:06.794253+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 7, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T16:01:14.963100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-17T16:01:17.236630+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.019999980926514, 'score_stable': 4.019999980926514, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-17T16:01:29.829066+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 19, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '53', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 35, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '282c4269-7e53-43fc-af4b-3b133d82837c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6809', 'nonce': 865094279}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6811', 'nonce': 865094279}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6815', 'nonce': 865094279}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 865094279}, {'type': 'v1', 'addr': '172.21.15.6:6813', 'nonce': 865094279}]}, 'public_addr': '172.21.15.6:6809/865094279', 'cluster_addr': '172.21.15.6:6811/865094279', 'heartbeat_back_addr': '172.21.15.6:6815/865094279', 'heartbeat_front_addr': '172.21.15.6:6813/865094279', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '66e20d10-2bb0-4130-93a5-45bcdd2de605', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6801', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 1460756480}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6805', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 1460756480}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6813', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 1460756480}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6809', 'nonce': 1460756480}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 1460756480}]}, 'public_addr': '172.21.15.50:6803/1460756480', 'cluster_addr': '172.21.15.50:6807/1460756480', 'heartbeat_back_addr': '172.21.15.50:6815/1460756480', 'heartbeat_front_addr': '172.21.15.50:6811/1460756480', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c6ec9e20-133a-471b-b63c-6453a7bde86c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6816', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6817', 'nonce': 1905413450}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6818', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6819', 'nonce': 1905413450}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6822', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6823', 'nonce': 1905413450}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6820', 'nonce': 1905413450}, {'type': 'v1', 'addr': '172.21.15.169:6821', 'nonce': 1905413450}]}, 'public_addr': '172.21.15.169:6817/1905413450', 'cluster_addr': '172.21.15.169:6819/1905413450', 'heartbeat_back_addr': '172.21.15.169:6823/1905413450', 'heartbeat_front_addr': '172.21.15.169:6821/1905413450', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3491a731-ca96-4981-be73-e005e00c315f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6810', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6811', 'nonce': 2918557179}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6812', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6813', 'nonce': 2918557179}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6816', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6817', 'nonce': 2918557179}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6814', 'nonce': 2918557179}, {'type': 'v1', 'addr': '172.21.15.181:6815', 'nonce': 2918557179}]}, 'public_addr': '172.21.15.181:6811/2918557179', 'cluster_addr': '172.21.15.181:6813/2918557179', 'heartbeat_back_addr': '172.21.15.181:6817/2918557179', 'heartbeat_front_addr': '172.21.15.181:6815/2918557179', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '3bdfaa34-bc10-4cec-b5c2-86e05d444ffd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6800', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6801', 'nonce': 2257685015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6803', 'nonce': 2257685015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6807', 'nonce': 2257685015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 2257685015}, {'type': 'v1', 'addr': '172.21.15.6:6805', 'nonce': 2257685015}]}, 'public_addr': '172.21.15.6:6801/2257685015', 'cluster_addr': '172.21.15.6:6803/2257685015', 'heartbeat_back_addr': '172.21.15.6:6807/2257685015', 'heartbeat_front_addr': '172.21.15.6:6805/2257685015', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 2399951648}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 2399951648}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 2399951648}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 2399951648}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 2399951648}]}, 'public_addr': '172.21.15.50:6825/2399951648', 'cluster_addr': '172.21.15.50:6827/2399951648', 'heartbeat_back_addr': '172.21.15.50:6831/2399951648', 'heartbeat_front_addr': '172.21.15.50:6829/2399951648', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ddd3edea-260a-4ec9-8533-841ed933fdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6824', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6825', 'nonce': 1225429423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6826', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6827', 'nonce': 1225429423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6830', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6831', 'nonce': 1225429423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6828', 'nonce': 1225429423}, {'type': 'v1', 'addr': '172.21.15.169:6829', 'nonce': 1225429423}]}, 'public_addr': '172.21.15.169:6825/1225429423', 'cluster_addr': '172.21.15.169:6827/1225429423', 'heartbeat_back_addr': '172.21.15.169:6831/1225429423', 'heartbeat_front_addr': '172.21.15.169:6829/1225429423', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7afc1a18-98c2-4a58-a09d-1b4f474d4dc4', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 57, 'down_at': 65, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6818', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6819', 'nonce': 3415295348}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6820', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6821', 'nonce': 3415295348}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6824', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6825', 'nonce': 3415295348}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6822', 'nonce': 3415295348}, {'type': 'v1', 'addr': '172.21.15.181:6823', 'nonce': 3415295348}]}, 'public_addr': '172.21.15.181:6819/3415295348', 'cluster_addr': '172.21.15.181:6821/3415295348', 'heartbeat_back_addr': '172.21.15.181:6825/3415295348', 'heartbeat_front_addr': '172.21.15.181:6823/3415295348', 'state': ['exists']}, {'osd': 8, 'uuid': '3e45f80f-856b-4d3f-b002-255f16a14f6b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6824', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6825', 'nonce': 2216439459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6826', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6827', 'nonce': 2216439459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6830', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6831', 'nonce': 2216439459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6828', 'nonce': 2216439459}, {'type': 'v1', 'addr': '172.21.15.6:6829', 'nonce': 2216439459}]}, 'public_addr': '172.21.15.6:6825/2216439459', 'cluster_addr': '172.21.15.6:6827/2216439459', 'heartbeat_back_addr': '172.21.15.6:6831/2216439459', 'heartbeat_front_addr': '172.21.15.6:6829/2216439459', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'b415c423-24de-4d86-bec4-ea330364ff96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 25, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 1727176716}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 1727176716}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 1727176716}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 1727176716}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 1727176716}]}, 'public_addr': '172.21.15.50:6817/1727176716', 'cluster_addr': '172.21.15.50:6819/1727176716', 'heartbeat_back_addr': '172.21.15.50:6823/1727176716', 'heartbeat_front_addr': '172.21.15.50:6821/1727176716', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': 'e471e1cb-9c0b-4c99-8622-59eb7956770d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6808', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6809', 'nonce': 3604183526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6810', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6811', 'nonce': 3604183526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6814', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6815', 'nonce': 3604183526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6812', 'nonce': 3604183526}, {'type': 'v1', 'addr': '172.21.15.169:6813', 'nonce': 3604183526}]}, 'public_addr': '172.21.15.169:6809/3604183526', 'cluster_addr': '172.21.15.169:6811/3604183526', 'heartbeat_back_addr': '172.21.15.169:6815/3604183526', 'heartbeat_front_addr': '172.21.15.169:6813/3604183526', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'c2bf4255-7ce1-4673-bedf-fd055d841fac', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6826', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6827', 'nonce': 3420013159}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6828', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6829', 'nonce': 3420013159}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6832', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6833', 'nonce': 3420013159}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6830', 'nonce': 3420013159}, {'type': 'v1', 'addr': '172.21.15.181:6831', 'nonce': 3420013159}]}, 'public_addr': '172.21.15.181:6827/3420013159', 'cluster_addr': '172.21.15.181:6829/3420013159', 'heartbeat_back_addr': '172.21.15.181:6833/3420013159', 'heartbeat_front_addr': '172.21.15.181:6831/3420013159', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '4c16f3ee-f427-41d5-b0ba-9e676ddbe032', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 23, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6817', 'nonce': 2310487797}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6818', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6819', 'nonce': 2310487797}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6822', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6823', 'nonce': 2310487797}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6820', 'nonce': 2310487797}, {'type': 'v1', 'addr': '172.21.15.6:6821', 'nonce': 2310487797}]}, 'public_addr': '172.21.15.6:6817/2310487797', 'cluster_addr': '172.21.15.6:6819/2310487797', 'heartbeat_back_addr': '172.21.15.6:6823/2310487797', 'heartbeat_front_addr': '172.21.15.6:6821/2310487797', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '0a4386aa-1a59-44c8-8d1e-7d43bf10de2e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6800', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6802', 'nonce': 1210437211}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6806', 'nonce': 1210437211}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6814', 'nonce': 1210437211}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 1210437211}, {'type': 'v1', 'addr': '172.21.15.50:6810', 'nonce': 1210437211}]}, 'public_addr': '172.21.15.50:6802/1210437211', 'cluster_addr': '172.21.15.50:6806/1210437211', 'heartbeat_back_addr': '172.21.15.50:6814/1210437211', 'heartbeat_front_addr': '172.21.15.50:6810/1210437211', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '4c54e35c-e226-4d8c-bad0-51dd1cd95fe0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 53, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6800', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6801', 'nonce': 4170280864}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6802', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6803', 'nonce': 4170280864}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6806', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6807', 'nonce': 4170280864}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.169:6804', 'nonce': 4170280864}, {'type': 'v1', 'addr': '172.21.15.169:6805', 'nonce': 4170280864}]}, 'public_addr': '172.21.15.169:6801/4170280864', 'cluster_addr': '172.21.15.169:6803/4170280864', 'heartbeat_back_addr': '172.21.15.169:6807/4170280864', 'heartbeat_front_addr': '172.21.15.169:6805/4170280864', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': '0281e750-f74a-4760-bbef-fbfa4127d06b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6802', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6803', 'nonce': 2815528414}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6804', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6805', 'nonce': 2815528414}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6808', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6809', 'nonce': 2815528414}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6806', 'nonce': 2815528414}, {'type': 'v1', 'addr': '172.21.15.181:6807', 'nonce': 2815528414}]}, 'public_addr': '172.21.15.181:6803/2815528414', 'cluster_addr': '172.21.15.181:6805/2815528414', 'heartbeat_back_addr': '172.21.15.181:6809/2815528414', 'heartbeat_front_addr': '172.21.15.181:6807/2815528414', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.014883+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.677746+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.838957+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.321698+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:02:25.617972+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:02:25.371442+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.517939+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.664321+0000', 'dead_epoch': 65}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:02:26.141597+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:02:25.015167+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.124128+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.554475+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:13.868839+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.912548+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:01:12.882601+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-17T16:02:26.664633+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.7', 'osds': [13, 15]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}, {'begin': 8, 'length': 3}, {'begin': 13, 'length': 3}, {'begin': 20, 'length': 2}, {'begin': 23, 'length': 1}, {'begin': 26, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-17T16:02:27.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:02:27.918 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:27.913+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:27.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:28.016 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:28.013+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:28.021 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.021 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:28.117 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:28.114+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:28.141 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.141 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.150 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:28.234 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:28.231+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:28.265 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:28.265 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:02:28.277 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-17T16:02:27.481835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117178931,"num_objects":154,"num_object_clones":100,"num_object_copies":1058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":6,"num_read":320,"num_read_kb":80956,"num_write":596,"num_write_kb":89682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1367,"ondisk_log_size":1367,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":670692,"kb_used_data":200780,"kb_used_omap":24,"kb_used_meta":469543,"kb_avail":1509278748,"statfs":{"total":1546188226560,"available":1545501437952,"internally_reserved":0,"allocated":205598720,"data_stored":300682136,"data_compressed":3026365,"data_compressed_allocated":4104192,"data_compressed_original":9084928,"omap_allocated":25519,"internal_metadata":480812113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21839872,"num_objects":14,"num_object_clones":14,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":-3,"num_read":24,"num_read_kb":2613,"num_write":70,"num_write_kb":20144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.027025"},"pg_stats":[{"pgid":"3.19","version":"84'53","reported_seq":343,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464449+0000","last_change":"2024-09-17T16:02:22.124783+0000","last_active":"2024-09-17T16:02:24.464449+0000","last_peered":"2024-09-17T16:02:24.464449+0000","last_clean":"2024-09-17T16:02:24.464449+0000","last_became_active":"2024-09-17T16:01:52.357491+0000","last_became_peered":"2024-09-17T16:01:52.357491+0000","last_unstale":"2024-09-17T16:02:24.464449+0000","last_undegraded":"2024-09-17T16:02:24.464449+0000","last_fullsized":"2024-09-17T16:02:24.464449+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:01:59.252035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.252035+0000","objects_scrubbed":4,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:24.337213+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.68151890599999998,"stat_sum":{"num_bytes":6275072,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":772,"num_write":10,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.18","version":"83'64","reported_seq":344,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.463784+0000","last_change":"2024-09-17T16:02:21.450455+0000","last_active":"2024-09-17T16:02:24.463784+0000","last_peered":"2024-09-17T16:02:24.463784+0000","last_clean":"2024-09-17T16:02:24.463784+0000","last_became_active":"2024-09-17T16:01:53.177881+0000","last_became_peered":"2024-09-17T16:01:53.177881+0000","last_unstale":"2024-09-17T16:02:24.463784+0000","last_undegraded":"2024-09-17T16:02:24.463784+0000","last_fullsized":"2024-09-17T16:02:24.463784+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'31","last_scrub_stamp":"2024-09-17T16:02:00.137409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.137409+0000","objects_scrubbed":2,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:29.319264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012806725999999999,"stat_sum":{"num_bytes":5603328,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":13,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.17","version":"84'44","reported_seq":288,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.460937+0000","last_change":"2024-09-17T16:02:04.352378+0000","last_active":"2024-09-17T16:02:24.460937+0000","last_peered":"2024-09-17T16:02:24.460937+0000","last_clean":"2024-09-17T16:02:02.351935+0000","last_became_active":"2024-09-17T16:02:04.352378+0000","last_became_peered":"2024-09-17T16:02:04.352378+0000","last_unstale":"2024-09-17T16:02:24.460937+0000","last_undegraded":"2024-09-17T16:02:04.343159+0000","last_fullsized":"2024-09-17T16:02:04.342807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'27","last_scrub_stamp":"2024-09-17T16:02:02.077678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.077678+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:29.310623+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.027309317,"stat_sum":{"num_bytes":5914624,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":819,"num_write":11,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.16","version":"84'34","reported_seq":290,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.461631+0000","last_change":"2024-09-17T16:02:04.351696+0000","last_active":"2024-09-17T16:02:24.461631+0000","last_peered":"2024-09-17T16:02:24.461631+0000","last_clean":"2024-09-17T16:02:02.354687+0000","last_became_active":"2024-09-17T16:02:04.351696+0000","last_became_peered":"2024-09-17T16:02:04.351696+0000","last_unstale":"2024-09-17T16:02:24.461631+0000","last_undegraded":"2024-09-17T16:02:04.341016+0000","last_fullsized":"2024-09-17T16:02:04.340608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'16","last_scrub_stamp":"2024-09-17T16:02:00.050173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.050173+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:14.978114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01423026,"stat_sum":{"num_bytes":6881280,"num_objects":7,"num_object_clones":5,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":7052,"num_write":10,"num_write_kb":3232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":7}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.15","version":"84'30","reported_seq":318,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.465715+0000","last_change":"2024-09-17T16:02:21.447459+0000","last_active":"2024-09-17T16:02:24.465715+0000","last_peered":"2024-09-17T16:02:24.465715+0000","last_clean":"2024-09-17T16:02:24.465715+0000","last_became_active":"2024-09-17T16:01:53.142297+0000","last_became_peered":"2024-09-17T16:01:53.142297+0000","last_unstale":"2024-09-17T16:02:24.465715+0000","last_undegraded":"2024-09-17T16:02:24.465715+0000","last_fullsized":"2024-09-17T16:02:24.465715+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'8","last_scrub_stamp":"2024-09-17T16:01:58.940426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.940426+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:04.522924+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0.014573063000000001,"stat_sum":{"num_bytes":2834432,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.14","version":"83'32","reported_seq":322,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-17T16:02:23.456208+0000","last_change":"2024-09-17T16:02:21.442226+0000","last_active":"2024-09-17T16:02:23.456208+0000","last_peered":"2024-09-17T16:02:23.456208+0000","last_clean":"2024-09-17T16:02:23.456208+0000","last_became_active":"2024-09-17T16:01:52.360122+0000","last_became_peered":"2024-09-17T16:01:52.360122+0000","last_unstale":"2024-09-17T16:02:23.456208+0000","last_undegraded":"2024-09-17T16:02:23.456208+0000","last_fullsized":"2024-09-17T16:02:23.456208+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:58.496684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.496684+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:01.254758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.009563518,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.460596+0000","last_change":"2024-09-17T16:01:58.347762+0000","last_active":"2024-09-17T16:02:24.460596+0000","last_peered":"2024-09-17T16:02:24.460596+0000","last_clean":"2024-09-17T16:02:24.460596+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:02:24.460596+0000","last_undegraded":"2024-09-17T16:02:24.460596+0000","last_fullsized":"2024-09-17T16:02:24.460596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:30.740032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"85'24","reported_seq":272,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.466804+0000","last_change":"2024-09-17T16:02:04.349120+0000","last_active":"2024-09-17T16:02:24.466804+0000","last_peered":"2024-09-17T16:02:24.466804+0000","last_clean":"2024-09-17T16:02:02.355614+0000","last_became_active":"2024-09-17T16:02:04.349120+0000","last_became_peered":"2024-09-17T16:02:04.349120+0000","last_unstale":"2024-09-17T16:02:24.466804+0000","last_undegraded":"2024-09-17T16:02:04.340835+0000","last_fullsized":"2024-09-17T16:02:04.340587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:02:01.398192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.398192+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:30.947896+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0.018760373,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":964,"num_write":9,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":135,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.457482+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:02:24.457482+0000","last_peered":"2024-09-17T16:02:24.457482+0000","last_clean":"2024-09-17T16:02:24.457482+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:02:24.457482+0000","last_undegraded":"2024-09-17T16:02:24.457482+0000","last_fullsized":"2024-09-17T16:02:24.457482+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"72'22","reported_seq":256,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.460978+0000","last_change":"2024-09-17T16:02:04.349605+0000","last_active":"2024-09-17T16:02:24.460978+0000","last_peered":"2024-09-17T16:02:24.460978+0000","last_clean":"2024-09-17T16:02:03.036314+0000","last_became_active":"2024-09-17T16:02:04.349605+0000","last_became_peered":"2024-09-17T16:02:04.349605+0000","last_unstale":"2024-09-17T16:02:24.460978+0000","last_undegraded":"2024-09-17T16:02:04.340916+0000","last_fullsized":"2024-09-17T16:02:04.340610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01416717,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":15,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":135,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.483022+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:02:24.483022+0000","last_peered":"2024-09-17T16:02:24.483022+0000","last_clean":"2024-09-17T16:02:24.483022+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:02:24.483022+0000","last_undegraded":"2024-09-17T16:02:24.483022+0000","last_fullsized":"2024-09-17T16:02:24.483022+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:19.517195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":293,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-17T16:02:23.456325+0000","last_change":"2024-09-17T16:02:21.433572+0000","last_active":"2024-09-17T16:02:23.456325+0000","last_peered":"2024-09-17T16:02:23.456325+0000","last_clean":"2024-09-17T16:02:23.456325+0000","last_became_active":"2024-09-17T16:01:52.351079+0000","last_became_peered":"2024-09-17T16:01:52.351079+0000","last_unstale":"2024-09-17T16:02:23.456325+0000","last_undegraded":"2024-09-17T16:02:23.456325+0000","last_fullsized":"2024-09-17T16:02:23.456325+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:59.453974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.453974+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.791395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00084878899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":133,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-17T16:02:23.456403+0000","last_change":"2024-09-17T16:01:20.762172+0000","last_active":"2024-09-17T16:02:23.456403+0000","last_peered":"2024-09-17T16:02:23.456403+0000","last_clean":"2024-09-17T16:02:23.456403+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:02:23.456403+0000","last_undegraded":"2024-09-17T16:02:23.456403+0000","last_fullsized":"2024-09-17T16:02:23.456403+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.827415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"83'24","reported_seq":306,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.466026+0000","last_change":"2024-09-17T16:02:21.442931+0000","last_active":"2024-09-17T16:02:24.466026+0000","last_peered":"2024-09-17T16:02:24.466026+0000","last_clean":"2024-09-17T16:02:24.466026+0000","last_became_active":"2024-09-17T16:01:52.345743+0000","last_became_peered":"2024-09-17T16:01:52.345743+0000","last_unstale":"2024-09-17T16:02:24.466026+0000","last_undegraded":"2024-09-17T16:02:24.466026+0000","last_fullsized":"2024-09-17T16:02:24.466026+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'8","last_scrub_stamp":"2024-09-17T16:01:59.457065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.457065+0000","objects_scrubbed":2,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:07.895492+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0099036009999999997,"stat_sum":{"num_bytes":344064,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.3","version":"84'39","reported_seq":322,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464594+0000","last_change":"2024-09-17T16:02:21.439216+0000","last_active":"2024-09-17T16:02:24.464594+0000","last_peered":"2024-09-17T16:02:24.464594+0000","last_clean":"2024-09-17T16:02:24.464594+0000","last_became_active":"2024-09-17T16:01:52.349931+0000","last_became_peered":"2024-09-17T16:01:52.349931+0000","last_unstale":"2024-09-17T16:02:24.464594+0000","last_undegraded":"2024-09-17T16:02:24.464594+0000","last_fullsized":"2024-09-17T16:02:24.464594+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'18","last_scrub_stamp":"2024-09-17T16:01:59.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.099258+0000","objects_scrubbed":4,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:24.478344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0099998980000000001,"stat_sum":{"num_bytes":1212416,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":62,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":138,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.465732+0000","last_change":"2024-09-17T16:01:20.767827+0000","last_active":"2024-09-17T16:02:24.465732+0000","last_peered":"2024-09-17T16:02:24.465732+0000","last_clean":"2024-09-17T16:02:24.465732+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:02:24.465732+0000","last_undegraded":"2024-09-17T16:02:24.465732+0000","last_fullsized":"2024-09-17T16:02:24.465732+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:33.840558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"83'80","reported_seq":351,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.461115+0000","last_change":"2024-09-17T16:02:21.456405+0000","last_active":"2024-09-17T16:02:24.461115+0000","last_peered":"2024-09-17T16:02:24.461115+0000","last_clean":"2024-09-17T16:02:24.461115+0000","last_became_active":"2024-09-17T16:01:52.345493+0000","last_became_peered":"2024-09-17T16:01:52.345493+0000","last_unstale":"2024-09-17T16:02:24.461115+0000","last_undegraded":"2024-09-17T16:02:24.461115+0000","last_fullsized":"2024-09-17T16:02:24.461115+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'27","last_scrub_stamp":"2024-09-17T16:02:00.289587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.289587+0000","objects_scrubbed":4,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:22.372765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013542337999999999,"stat_sum":{"num_bytes":2850816,"num_objects":8,"num_object_clones":6,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":3048,"num_write":9,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":135,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464105+0000","last_change":"2024-09-17T16:01:21.267857+0000","last_active":"2024-09-17T16:02:24.464105+0000","last_peered":"2024-09-17T16:02:24.464105+0000","last_clean":"2024-09-17T16:02:24.464105+0000","last_became_active":"2024-09-17T16:01:18.759808+0000","last_became_peered":"2024-09-17T16:01:18.759808+0000","last_unstale":"2024-09-17T16:02:24.464105+0000","last_undegraded":"2024-09-17T16:02:24.464105+0000","last_fullsized":"2024-09-17T16:02:24.464105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:29.387981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039068600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.1","version":"72'19","reported_seq":264,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.460891+0000","last_change":"2024-09-17T16:02:04.352734+0000","last_active":"2024-09-17T16:02:24.460891+0000","last_peered":"2024-09-17T16:02:24.460891+0000","last_clean":"2024-09-17T16:02:02.350917+0000","last_became_active":"2024-09-17T16:02:04.352734+0000","last_became_peered":"2024-09-17T16:02:04.352734+0000","last_unstale":"2024-09-17T16:02:24.460891+0000","last_undegraded":"2024-09-17T16:02:04.341137+0000","last_fullsized":"2024-09-17T16:02:04.340823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.086155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.086155+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:21.279762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014571756,"stat_sum":{"num_bytes":2080768,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":202,"num_write":3,"num_write_kb":800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,2147483647],"acting":[13,4,3,11,10,2,2147483647],"avail_no_missing":["13(0)","2(5)","3(2)","4(1)","10(4)","11(3)"],"object_location_counts":[{"shards":"2(5),3(2),4(1),10(4),11(3),13(0)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":136,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.465528+0000","last_change":"2024-09-17T16:01:20.767402+0000","last_active":"2024-09-17T16:02:24.465528+0000","last_peered":"2024-09-17T16:02:24.465528+0000","last_clean":"2024-09-17T16:02:24.465528+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:02:24.465528+0000","last_undegraded":"2024-09-17T16:02:24.465528+0000","last_fullsized":"2024-09-17T16:02:24.465528+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.041494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"79'47","reported_seq":291,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.465295+0000","last_change":"2024-09-17T16:02:04.349933+0000","last_active":"2024-09-17T16:02:24.465295+0000","last_peered":"2024-09-17T16:02:24.465295+0000","last_clean":"2024-09-17T16:02:02.354956+0000","last_became_active":"2024-09-17T16:02:04.349933+0000","last_became_peered":"2024-09-17T16:02:04.349933+0000","last_unstale":"2024-09-17T16:02:24.465295+0000","last_undegraded":"2024-09-17T16:02:04.341956+0000","last_fullsized":"2024-09-17T16:02:04.341160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'35","last_scrub_stamp":"2024-09-17T16:02:01.159864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.159864+0000","objects_scrubbed":7,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:23.420419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023769390000000001,"stat_sum":{"num_bytes":4030464,"num_objects":10,"num_object_clones":7,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":138,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,2147483647],"acting":[11,2,0,4,8,3,2147483647],"avail_no_missing":["11(0)","0(2)","2(1)","3(5)","4(3)","8(4)"],"object_location_counts":[{"shards":"0(2),2(1),3(5),4(3),8(4),11(0)","objects":10}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":136,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464927+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:02:24.464927+0000","last_peered":"2024-09-17T16:02:24.464927+0000","last_clean":"2024-09-17T16:02:24.464927+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:02:24.464927+0000","last_undegraded":"2024-09-17T16:02:24.464927+0000","last_fullsized":"2024-09-17T16:02:24.464927+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:38.436314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":270,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464324+0000","last_change":"2024-09-17T16:01:16.943096+0000","last_active":"2024-09-17T16:02:24.464324+0000","last_peered":"2024-09-17T16:02:24.464324+0000","last_clean":"2024-09-17T16:02:24.464324+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:02:24.464324+0000","last_undegraded":"2024-09-17T16:02:24.464324+0000","last_fullsized":"2024-09-17T16:02:24.464324+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:01:15.727345+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:36.962646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"84'54","reported_seq":332,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464095+0000","last_change":"2024-09-17T16:02:21.443715+0000","last_active":"2024-09-17T16:02:24.464095+0000","last_peered":"2024-09-17T16:02:24.464095+0000","last_clean":"2024-09-17T16:02:24.464095+0000","last_became_active":"2024-09-17T16:01:52.347600+0000","last_became_peered":"2024-09-17T16:01:52.347600+0000","last_unstale":"2024-09-17T16:02:24.464095+0000","last_undegraded":"2024-09-17T16:02:24.464095+0000","last_fullsized":"2024-09-17T16:02:24.464095+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'33","last_scrub_stamp":"2024-09-17T16:02:02.109565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.109565+0000","objects_scrubbed":3,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:17.018990+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0058930329999999998,"stat_sum":{"num_bytes":4063232,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":10525,"num_write":7,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.9","version":"83'51","reported_seq":328,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.461182+0000","last_change":"2024-09-17T16:02:21.456475+0000","last_active":"2024-09-17T16:02:24.461182+0000","last_peered":"2024-09-17T16:02:24.461182+0000","last_clean":"2024-09-17T16:02:24.461182+0000","last_became_active":"2024-09-17T16:01:52.347076+0000","last_became_peered":"2024-09-17T16:01:52.347076+0000","last_unstale":"2024-09-17T16:02:24.461182+0000","last_undegraded":"2024-09-17T16:02:24.461182+0000","last_fullsized":"2024-09-17T16:02:24.461182+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:02:01.251580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.251580+0000","objects_scrubbed":2,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:20.424120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.013631613000000001,"stat_sum":{"num_bytes":2949120,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":7,"num_read_kb":451,"num_write":4,"num_write_kb":1872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,10],"acting":[14,11,13,15,4,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.a","version":"86'56","reported_seq":281,"reported_epoch":88,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:26.481472+0000","last_change":"2024-09-17T16:02:04.994580+0000","last_active":"2024-09-17T16:02:26.481472+0000","last_peered":"2024-09-17T16:02:26.481472+0000","last_clean":"2024-09-17T16:02:02.349037+0000","last_became_active":"2024-09-17T16:02:04.994580+0000","last_became_peered":"2024-09-17T16:02:04.994580+0000","last_unstale":"2024-09-17T16:02:26.481472+0000","last_undegraded":"2024-09-17T16:02:04.341211+0000","last_fullsized":"2024-09-17T16:02:04.340891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:39.808989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081311999999999,"stat_sum":{"num_bytes":12795904,"num_objects":10,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":18,"num_read_kb":20811,"num_write":28,"num_write_kb":12496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,2147483647,3,14,6,8],"acting":[12,4,2147483647,3,14,6,8],"avail_no_missing":["12(0)","3(3)","4(1)","6(5)","8(6)","14(4)"],"object_location_counts":[{"shards":"3(3),4(1),6(5),8(6),12(0),14(4)","objects":10}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.b","version":"84'24","reported_seq":252,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.461296+0000","last_change":"2024-09-17T16:02:04.347340+0000","last_active":"2024-09-17T16:02:24.461296+0000","last_peered":"2024-09-17T16:02:24.461296+0000","last_clean":"2024-09-17T16:02:02.337229+0000","last_became_active":"2024-09-17T16:02:04.347340+0000","last_became_peered":"2024-09-17T16:02:04.347340+0000","last_unstale":"2024-09-17T16:02:24.461296+0000","last_undegraded":"2024-09-17T16:02:04.340875+0000","last_fullsized":"2024-09-17T16:02:04.340555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:35.445238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010428939999999999,"stat_sum":{"num_bytes":3063808,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":20,"num_read_kb":1683,"num_write":13,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,2147483647],"acting":[13,1,11,15,6,4,2147483647],"avail_no_missing":["13(0)","1(1)","4(5)","6(4)","11(2)","15(3)"],"object_location_counts":[{"shards":"1(1),4(5),6(4),11(2),13(0),15(3)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.c","version":"84'56","reported_seq":306,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464381+0000","last_change":"2024-09-17T16:02:21.443222+0000","last_active":"2024-09-17T16:02:24.464381+0000","last_peered":"2024-09-17T16:02:24.464381+0000","last_clean":"2024-09-17T16:02:24.464381+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:02:24.464381+0000","last_undegraded":"2024-09-17T16:02:24.464381+0000","last_fullsized":"2024-09-17T16:02:24.464381+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:50.825970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070758050000000001,"stat_sum":{"num_bytes":3473408,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":2532,"num_write":37,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.d","version":"86'201","reported_seq":466,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-09-17T16:02:26.481235+0000","last_change":"2024-09-17T16:02:21.954214+0000","last_active":"2024-09-17T16:02:26.481235+0000","last_peered":"2024-09-17T16:02:26.481235+0000","last_clean":"2024-09-17T16:02:26.481235+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:02:26.481235+0000","last_undegraded":"2024-09-17T16:02:26.481235+0000","last_fullsized":"2024-09-17T16:02:26.481235+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:57.039261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51541643699999995,"stat_sum":{"num_bytes":23166976,"num_objects":23,"num_object_clones":17,"num_object_copies":161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":59,"num_read_kb":18326,"num_write":95,"num_write_kb":24496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.e","version":"84'105","reported_seq":365,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.458391+0000","last_change":"2024-09-17T16:02:21.448920+0000","last_active":"2024-09-17T16:02:24.458391+0000","last_peered":"2024-09-17T16:02:24.458391+0000","last_clean":"2024-09-17T16:02:24.458391+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:02:24.458391+0000","last_undegraded":"2024-09-17T16:02:24.458391+0000","last_fullsized":"2024-09-17T16:02:24.458391+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:58.085174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016340574,"stat_sum":{"num_bytes":11157504,"num_objects":12,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":32,"num_read_kb":11720,"num_write":81,"num_write_kb":8480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]},{"pgid":"3.f","version":"73'39","reported_seq":253,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.464572+0000","last_change":"2024-09-17T16:02:04.347090+0000","last_active":"2024-09-17T16:02:24.464572+0000","last_peered":"2024-09-17T16:02:24.464572+0000","last_clean":"2024-09-17T16:02:02.358329+0000","last_became_active":"2024-09-17T16:02:04.347090+0000","last_became_peered":"2024-09-17T16:02:04.347090+0000","last_unstale":"2024-09-17T16:02:24.464572+0000","last_undegraded":"2024-09-17T16:02:04.341882+0000","last_fullsized":"2024-09-17T16:02:04.341582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:47.856590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015713103999999999,"stat_sum":{"num_bytes":2605056,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":13,"num_read_kb":368,"num_write":36,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,2147483647,10,1],"acting":[0,2,14,11,2147483647,10,1],"avail_no_missing":["0(0)","1(6)","2(1)","10(5)","11(3)","14(2)"],"object_location_counts":[{"shards":"0(0),1(6),2(1),10(5),11(3),14(2)","objects":7}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.10","version":"77'43","reported_seq":165,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.483007+0000","last_change":"2024-09-17T16:02:04.348295+0000","last_active":"2024-09-17T16:02:24.483007+0000","last_peered":"2024-09-17T16:02:24.483007+0000","last_clean":"2024-09-17T16:02:02.344595+0000","last_became_active":"2024-09-17T16:02:04.348295+0000","last_became_peered":"2024-09-17T16:02:04.348295+0000","last_unstale":"2024-09-17T16:02:24.483007+0000","last_undegraded":"2024-09-17T16:02:04.341600+0000","last_fullsized":"2024-09-17T16:02:04.341131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'31","last_scrub_stamp":"2024-09-17T16:01:58.728617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.728617+0000","objects_scrubbed":3,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:12.370188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.16505651299999999,"stat_sum":{"num_bytes":2916352,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,8,15,5,9,0],"acting":[2147483647,6,8,15,5,9,0],"avail_no_missing":["6(1)","0(6)","5(4)","8(2)","9(5)","15(3)"],"object_location_counts":[{"shards":"0(6),5(4),6(1),8(2),9(5),15(3)","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"76'18","reported_seq":147,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.553970+0000","last_change":"2024-09-17T16:02:04.349949+0000","last_active":"2024-09-17T16:02:24.553970+0000","last_peered":"2024-09-17T16:02:24.553970+0000","last_clean":"2024-09-17T16:02:02.345085+0000","last_became_active":"2024-09-17T16:02:04.349949+0000","last_became_peered":"2024-09-17T16:02:04.349949+0000","last_unstale":"2024-09-17T16:02:24.553970+0000","last_undegraded":"2024-09-17T16:02:04.342317+0000","last_fullsized":"2024-09-17T16:02:04.342004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.333371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.333371+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:19.071630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011690380000000001,"stat_sum":{"num_bytes":3538944,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":871,"num_write":4,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,2147483647,1],"acting":[10,4,0,3,5,2147483647,1],"avail_no_missing":["10(0)","0(2)","1(6)","3(3)","4(1)","5(4)"],"object_location_counts":[{"shards":"0(2),1(6),3(3),4(1),5(4),10(0)","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.12","version":"72'39","reported_seq":134,"reported_epoch":86,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:24.464015+0000","last_change":"2024-09-17T16:02:05.063706+0000","last_active":"2024-09-17T16:02:24.464015+0000","last_peered":"2024-09-17T16:02:24.464015+0000","last_clean":"2024-09-17T16:02:02.344908+0000","last_became_active":"2024-09-17T16:02:05.063706+0000","last_became_peered":"2024-09-17T16:02:05.063706+0000","last_unstale":"2024-09-17T16:02:24.464015+0000","last_undegraded":"2024-09-17T16:02:04.342739+0000","last_fullsized":"2024-09-17T16:02:04.342330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'32","last_scrub_stamp":"2024-09-17T16:02:01.034730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.034730+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:16.801850+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0088005819999999995,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,6,12,2,2147483647],"acting":[15,8,5,6,12,2,2147483647],"avail_no_missing":["15(0)","2(5)","5(2)","6(3)","8(1)","12(4)"],"object_location_counts":[{"shards":"2(5),5(2),6(3),8(1),12(4),15(0)","objects":3}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.13","version":"83'38","reported_seq":328,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-17T16:02:24.464326+0000","last_change":"2024-09-17T16:02:21.438630+0000","last_active":"2024-09-17T16:02:24.464326+0000","last_peered":"2024-09-17T16:02:24.464326+0000","last_clean":"2024-09-17T16:02:24.464326+0000","last_became_active":"2024-09-17T16:01:52.366019+0000","last_became_peered":"2024-09-17T16:01:52.366019+0000","last_unstale":"2024-09-17T16:02:24.464326+0000","last_undegraded":"2024-09-17T16:02:24.464326+0000","last_fullsized":"2024-09-17T16:02:24.464326+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'16","last_scrub_stamp":"2024-09-17T16:02:00.060603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.060603+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:00.865290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0095342829999999993,"stat_sum":{"num_bytes":1277952,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":531,"num_write":1,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"3"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":116588544,"num_objects":150,"num_object_clones":100,"num_object_copies":1050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":6,"num_read":234,"num_read_kb":80883,"num_write":449,"num_write_kb":87008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192335872,"data_stored":290967552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1252,"ondisk_log_size":1252,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43012,"kb_used_data":11824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328828,"statfs":{"total":96636764160,"available":96592719872,"internally_reserved":0,"allocated":12107776,"data_stored":18101942,"data_compressed":195619,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280529,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37836,"kb_used_data":10800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334004,"statfs":{"total":96636764160,"available":96598020096,"internally_reserved":0,"allocated":11059200,"data_stored":14446152,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280530,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42156,"kb_used_data":15128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329684,"statfs":{"total":96636764160,"available":96593596416,"internally_reserved":0,"allocated":15491072,"data_stored":22521526,"data_compressed":195619,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313234,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45332,"kb_used_data":9988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326508,"statfs":{"total":96636764160,"available":96590344192,"internally_reserved":0,"allocated":10227712,"data_stored":13653466,"data_compressed":189379,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51140,"kb_used_data":19952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320700,"statfs":{"total":96636764160,"available":96584396800,"internally_reserved":0,"allocated":20430848,"data_stored":30633544,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280529,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53468,"kb_used_data":18120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94318372,"statfs":{"total":96636764160,"available":96582012928,"internally_reserved":0,"allocated":18554880,"data_stored":28417608,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36632,"kb_used_data":9600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335208,"statfs":{"total":96636764160,"available":96599252992,"internally_reserved":0,"allocated":9830400,"data_stored":13848680,"data_compressed":202997,"data_compressed_allocated":294912,"data_compressed_original":1028096,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280529,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33864,"kb_used_data":6836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337976,"statfs":{"total":96636764160,"available":96602087424,"internally_reserved":0,"allocated":7000064,"data_stored":10397146,"data_compressed":189379,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280529,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49024,"kb_used_data":17836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322816,"statfs":{"total":96636764160,"available":96586563584,"internally_reserved":0,"allocated":18264064,"data_stored":28016200,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280529,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41436,"kb_used_data":14408,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330404,"statfs":{"total":96636764160,"available":96594333696,"internally_reserved":0,"allocated":14753792,"data_stored":22042294,"data_compressed":195619,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,4,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280529,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41172,"kb_used_data":14136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330668,"statfs":{"total":96636764160,"available":96594604032,"internally_reserved":0,"allocated":14475264,"data_stored":21628598,"data_compressed":195619,"data_compressed_allocated":262144,"data_compressed_original":524288,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":5900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":6041600,"data_stored":8820876,"data_compressed":114770,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,5,6,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280529,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42224,"kb_used_data":15192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329616,"statfs":{"total":96636764160,"available":96593526784,"internally_reserved":0,"allocated":15556608,"data_stored":24030811,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280528,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33516,"kb_used_data":6484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338324,"statfs":{"total":96636764160,"available":96602443776,"internally_reserved":0,"allocated":6639616,"data_stored":8189421,"data_compressed":189379,"data_compressed_allocated":253952,"data_compressed_original":507904,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280529,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39480,"kb_used_data":8292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332360,"statfs":{"total":96636764160,"available":96596336640,"internally_reserved":0,"allocated":8491008,"data_stored":11148872,"data_compressed":192498,"data_compressed_allocated":258048,"data_compressed_original":516096,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280528,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47472,"kb_used_data":16284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324368,"statfs":{"total":96636764160,"available":96588152832,"internally_reserved":0,"allocated":16674816,"data_stored":24785000,"data_compressed":202997,"data_compressed_allocated":294912,"data_compressed_original":1028096,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19460096,"data_stored":29200384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9269248,"data_stored":12955648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8802304,"data_stored":12611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6565888,"data_stored":11030528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":17817600,"data_stored":27914240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":13770752,"data_stored":21323776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13492224,"data_stored":20910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5378048,"data_stored":8372224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15106048,"data_stored":23928832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":5677056,"data_stored":7491584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7520256,"data_stored":10440704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":15646720,"data_stored":23547904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":11124736,"data_stored":17383424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":17584128,"data_stored":26984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":15032320,"data_stored":23134208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10088448,"data_stored":13737984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:02:28.279 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [6, 11] 2024-09-17T16:02:28.279 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '6', '11'] 2024-09-17T16:02:28.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.1 6 11 2024-09-17T16:02:28.298 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.298 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:28.319 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:28.317+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:28.420 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:28.417+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:28.426 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 5 to smithi18136759-49 2024-09-17T16:02:28.458 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.458 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:28.483 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:316: done (2 left) 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:318: done (1 left) 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:319: copy_from oid 3 from oid 37 current snap is 35 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:320: read oid 46 snap -1 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:320: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:321: read oid 7 snap -1 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:321: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:322: read oid 32 snap -1 2024-09-17T16:02:28.488 INFO:tasks.rados.rados.0.smithi181.stdout:322: expect (ObjNum 117 snap 29 seq_num 117) 2024-09-17T16:02:28.489 INFO:tasks.rados.rados.0.smithi181.stdout:323: copy_from oid 36 from oid 25 current snap is 35 2024-09-17T16:02:28.489 INFO:tasks.rados.rados.0.smithi181.stdout:324: copy_from oid 35 from oid 11 current snap is 35 2024-09-17T16:02:28.489 INFO:tasks.rados.rados.0.smithi181.stdout:325: snap_create 2024-09-17T16:02:28.495 INFO:tasks.rados.rados.0.smithi181.stdout:321: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-17T16:02:28.495 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:28.497 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 76661 2024-09-17T16:02:28.497 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 19069 2024-09-17T16:02:28.498 INFO:tasks.rados.rados.0.smithi181.stdout:317: finishing write tid 6 to smithi18136759-49 2024-09-17T16:02:28.498 INFO:tasks.rados.rados.0.smithi181.stdout:317: oid 49 updating version 0 to 54 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:317: oid 49 version 54 is already newer than 53 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:317: oid 49 updating version 54 to 56 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:317: oid 49 updating version 56 to 57 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:317: oid 49 version 57 is already newer than 55 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 49 v 57 (ObjNum 129 snap 34 seq_num 129) dirty exists 2024-09-17T16:02:28.499 INFO:tasks.rados.rados.0.smithi181.stdout:317: left oid 49 (ObjNum 129 snap 34 seq_num 129) 2024-09-17T16:02:28.503 INFO:tasks.rados.rados.0.smithi181.stdout:320: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-17T16:02:28.506 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 476538 2024-09-17T16:02:28.509 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:28.518 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 255851 2024-09-17T16:02:28.522 INFO:tasks.rados.rados.0.smithi181.stdout:324: finishing copy_from racing read to smithi18136759-35 2024-09-17T16:02:28.522 INFO:tasks.rados.rados.0.smithi181.stdout:324: finishing copy_from to smithi18136759-35 2024-09-17T16:02:28.523 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 10 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T16:02:28.523 INFO:tasks.rados.rados.0.smithi181.stdout:323: finishing copy_from racing read to smithi18136759-36 2024-09-17T16:02:28.523 INFO:tasks.rados.rados.0.smithi181.stdout:323: finishing copy_from to smithi18136759-36 2024-09-17T16:02:28.523 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T16:02:28.542 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:28.539+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:28.580 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.580 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:28.622 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:28.620+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:28.722 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:28.721+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:28.726 INFO:tasks.rados.rados.0.smithi181.stdout:322: expect (ObjNum 117 snap 29 seq_num 117) 2024-09-17T16:02:28.727 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:28.736 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.737 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.745 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:28.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1960343 2024-09-17T16:02:28.773 INFO:tasks.rados.rados.0.smithi181.stdout:319: finishing copy_from racing read to smithi18136759-3 2024-09-17T16:02:28.823 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:28.821+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:28.853 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.853 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:28.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:28.924 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:28.923+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:28.940 INFO:tasks.rados.rados.0.smithi181.stdout:319: finishing copy_from to smithi18136759-3 2024-09-17T16:02:28.940 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 3 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T16:02:28.969 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:28.969 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:28.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:29.025 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:29.023+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:29.047 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:29.048 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:29.070 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:29.071 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:29.087 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:29.087 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:29.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:29.126 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:29.124+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:29.203 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:29.203 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:29.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:29.255 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:29.252+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:29.294 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:29.294 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:29.315 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:29.315 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:29.318 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:29.318 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:29.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:29.328 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:29.324+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:29.336 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:29.336 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:29.358 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:29.358 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:29.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:29.402 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:29.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:29.423 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:29.423 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:29.427 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:29.426+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:29.443 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:29.443 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:29.445 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:29.445 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:29.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:29.466 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:29.466 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:29.487 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:29.488 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:29.492 INFO:teuthology.orchestra.run.smithi006.stderr:set 2.1 pg_upmap mapping to [6,11] 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:317: done (7 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:319: done (6 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:320: done (5 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:321: done (4 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:322: done (3 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:323: done (2 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:324: done (1 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:325: done (0 left) 2024-09-17T16:02:29.497 INFO:tasks.rados.rados.0.smithi181.stdout:326: snap_remove snap 34 2024-09-17T16:02:29.510 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:29.511 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:29.529 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:29.526+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:29.532 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:29.533 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:29.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:29.530+0000 7f54b67fc640 0 -- 172.21.15.6:0/3088884372 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f54b8221a20 msgr2=0x7f54b8223e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:29.576 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:24 3.ds0 3:be8168ea:::smithi18136759-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.717577605999999, 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.04871255, 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.577 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 24 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6700000000000003e-07 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856438+0000", 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0089999999999999e-06 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856446+0000", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1240000000000005e-06 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856452+0000", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8490000000000002e-06 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858431+0000", 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019785380000000002 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858505+0000", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4269000000000003e-05 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876493+0000", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017988067999999999 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876576+0000", 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2981999999999999e-05 2024-09-17T16:02:29.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905147+0000", 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028571144 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:28 3.ds0 3:b6a72c22:::smithi18136759-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.717550785999997, 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043697429000000003, 2024-09-17T16:02:29.582 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 28 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856462+0000", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856586+0000", 2024-09-17T16:02:29.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012445300000000001 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5877000000000001e-05 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856609+0000", 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5869999999999998e-06 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858561+0000", 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019522739999999999 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858680+0000", 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118678 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859415+0000", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073567700000000001 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864061+0000", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046461560000000002 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900068+0000", 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036006330000000003 2024-09-17T16:02:29.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900159+0000", 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0970999999999993e-05 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:29 3.ds0 3:b6a72c22:::smithi18136759-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.717392163, 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043754779000000001, 2024-09-17T16:02:29.588 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 29 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856621+0000", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2099999999999996e-07 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856624+0000", 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7000000000000002e-06 2024-09-17T16:02:29.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856631+0000", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856638+0000", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859904+0000", 2024-09-17T16:02:29.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032664679999999998 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859980+0000", 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5711999999999996e-05 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860364+0000", 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038396600000000002 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864272+0000", 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003907888 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900273+0000", 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036001247 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900375+0000", 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101535 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.593 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:30 3.ds0 3:b6a72c22:::smithi18136759-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.717366337000001, 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.048780709999999998, 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.594 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 30 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4499999999999997e-07 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856657+0000", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856664+0000", 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.148e-06 2024-09-17T16:02:29.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856673+0000", 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.098e-06 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860632+0000", 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039593440000000001 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860703+0000", 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0197000000000006e-05 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900473+0000", 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039770158 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900637+0000", 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000164356 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905426+0000", 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047893579999999996 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:40 3.ds0 3:b22f7774:::smithi18136759-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.712362984000002, 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039805350000000003, 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.599 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 40 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861651+0000", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.525e-06 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861832+0000", 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018106599999999999 2024-09-17T16:02:29.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861846+0000", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4497e-05 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861854+0000", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0390000000000005e-06 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.791e-05 2024-09-17T16:02:29.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862054+0000", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014139799999999999 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862812+0000", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000758028 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867004+0000", 2024-09-17T16:02:29.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004192594 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901412+0000", 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034408162999999999 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901454+0000", 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2129999999999998e-05 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:41 3.ds0 3:b22f7774:::smithi18136759-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.712136678, 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039711297, 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 41 2024-09-17T16:02:29.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861876+0000", 2024-09-17T16:02:29.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4e-07 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861880+0000", 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861887+0000", 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0890000000000002e-06 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861893+0000", 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8649999999999998e-06 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863320+0000", 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001427212 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863412+0000", 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1540999999999997e-05 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863845+0000", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043304900000000002 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867006+0000", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031615369999999999 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901546+0000", 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034539746000000003 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901587+0000", 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0802e-05 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:42 3.ds0 3:b22f7774:::smithi18136759-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.712104183999998, 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043739746000000003, 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.610 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 42 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861909+0000", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9400000000000005e-07 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3019999999999999e-06 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861919+0000", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861925+0000", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8529999999999997e-06 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864137+0000", 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022118649999999999 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864205+0000", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8396e-05 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901627+0000", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037421487000000003 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901674+0000", 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7577000000000002e-05 2024-09-17T16:02:29.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905648+0000", 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039733169999999996 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:52 3.ds0 3:bf204631:::smithi18136759-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.703550495000002, 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031550519999999999, 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.615 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 52 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870463+0000", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.375e-06 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870715+0000", 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025214399999999999 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870733+0000", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7407000000000001e-05 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870749+0000", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6682000000000001e-05 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870815+0000", 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5939999999999995e-05 2024-09-17T16:02:29.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870975+0000", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015933899999999999 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.871691+0000", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071632 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.874021+0000", 2024-09-17T16:02:29.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023304419999999998 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901948+0000", 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027927041999999999 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902012+0000", 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3829000000000002e-05 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:54 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.703209375999997, 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035090831000000003, 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 54 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.621 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:29.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9800000000000004e-07 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870806+0000", 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3549999999999999e-06 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870813+0000", 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1539999999999996e-06 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870819+0000", 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0700000000000003e-06 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873071+0000", 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022519200000000001 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873154+0000", 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3746999999999996e-05 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902680+0000", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029525513 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902767+0000", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6837999999999998e-05 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905894+0000", 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031267360000000002 2024-09-17T16:02:29.625 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 46/32 ECSubWriteReply(tid=54, last_complete=46'16, committed=1, applied=1))", 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 44.635163161000001, 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62914934300000003, 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938850+0000", 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.06e-07 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938855+0000", 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0010000000000001e-06 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567797+0000", 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62894212500000002 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567814+0000", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6847e-05 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567931+0000", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011766500000000001 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567998+0000", 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6999e-05 2024-09-17T16:02:29.629 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:284 3.as0 3:5583c93f:::smithi18136759-40:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 43.640494183000001, 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.386964417, 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.630 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 284 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933519+0000", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933522+0000", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4010000000000001e-06 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933536+0000", 2024-09-17T16:02:29.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4269e-05 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933544+0000", 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.576e-06 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938388+0000", 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048441300000000003 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938478+0000", 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9975000000000005e-05 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939194+0000", 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071552499999999995 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944047+0000", 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048533990000000004 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320427+0000", 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37638003799999997 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320482+0000", 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5390000000000003e-05 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:358 3.ds0 3:b27fc74f:::smithi18136759-24:head [copy-from ver 28] snapc 11=[11,10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 36.197895488999997, 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.040742296999999997, 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 358 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.636 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:29.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376121+0000", 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1679999999999997e-06 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376135+0000", 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3054000000000001e-05 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376141+0000", 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.443e-06 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381260+0000", 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005118671 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381327+0000", 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7288999999999999e-05 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.392972+0000", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011645071999999999 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.396350+0000", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033780709999999999 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416805+0000", 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020454785999999999 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416859+0000", 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4045999999999997e-05 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 63/32 ECSubWriteReply(tid=124, last_complete=63'41, committed=1, applied=1))", 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.146683401000001, 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45322852000000002, 2024-09-17T16:02:29.641 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427336+0000", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4069999999999998e-06 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880036+0000", 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45270011199999999 2024-09-17T16:02:29.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880042+0000", 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7950000000000004e-06 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880109+0000", 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7659999999999999e-05 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880557+0000", 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044786499999999998 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:509 3.ds0 3:bf204631:::smithi18136759-18:head [setxattr _header (57) in=7b] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.199170926000001, 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91880832899999998, 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.645 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 509 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:29.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374842+0000", 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9999999999999995e-07 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374847+0000", 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.724e-06 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374855+0000", 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4400000000000005e-06 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374862+0000", 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377155+0000", 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022931840000000002 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377217+0000", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2512000000000001e-05 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377509+0000", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000291057 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.381196+0000", 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036871849999999999 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293578+0000", 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91238186700000001 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293650+0000", 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1950000000000001e-05 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.650 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:510 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.199141041000001, 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.92014742599999999, 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 510 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374872+0000", 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374875+0000", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.083e-06 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374883+0000", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374890+0000", 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9469999999999997e-06 2024-09-17T16:02:29.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378503+0000", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036131100000000001 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378624+0000", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121218 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294084+0000", 2024-09-17T16:02:29.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91545977899999997 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294190+0000", 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105797 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.295019+0000", 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082872900000000005 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 70/32 ECSubWriteReply(tid=171, last_complete=70'105, committed=1, applied=1))", 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.195438111000001, 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91541347900000003, 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378575+0000", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378579+0000", 2024-09-17T16:02:29.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0289999999999999e-06 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293447+0000", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91486866300000003 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293456+0000", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5040000000000002e-06 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293520+0000", 2024-09-17T16:02:29.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3893999999999995e-05 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293987+0000", 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000467725 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.659 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/32 ECSubWriteReply(tid=246, last_complete=81'169, committed=1, applied=1))", 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.118875965999999, 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.121075089, 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455137+0000", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0999999999999997e-07 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455142+0000", 2024-09-17T16:02:29.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9250000000000003e-06 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575575+0000", 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.120433441 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575581+0000", 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5720000000000002e-06 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575664+0000", 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3115000000000006e-05 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.576211+0000", 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054722600000000005 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 85/32 ECSubWriteReply(tid=275, last_complete=85'195, committed=1, applied=1))", 2024-09-17T16:02:29.663 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.1053423670000004, 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87648519400000002, 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3600000000000003e-07 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468673+0000", 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0400000000000001e-06 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344179+0000", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87550525700000004 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344191+0000", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2237e-05 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344247+0000", 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6391000000000003e-05 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.345155+0000", 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090763300000000003 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:768 3.as0 3:5ccfabf6:::smithi18136759-44:head [setxattr _header (57) in=7b] snapc 20=[20,1f,1e,1d,1c,19,18,16,10,c,6] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.0856584790000001, 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.801372309, 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.667 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 768 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5199999999999996e-07 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488358+0000", 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4479999999999999e-06 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488366+0000", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8380000000000002e-06 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488370+0000", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2420000000000002e-06 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492136+0000", 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037664959999999998 2024-09-17T16:02:29.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492251+0000", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114357 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492748+0000", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00049682299999999997 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.494817+0000", 2024-09-17T16:02:29.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002069286 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289552+0000", 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79473559500000002 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289726+0000", 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017347200000000001 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:769 3.as0 3:5ccfabf6:::smithi18136759-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.085635989, 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80260884799999999, 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 769 2024-09-17T16:02:29.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488377+0000", 2024-09-17T16:02:29.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7099999999999997e-07 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488380+0000", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7069999999999998e-06 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488386+0000", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4970000000000001e-06 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488389+0000", 2024-09-17T16:02:29.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9869999999999999e-06 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493011+0000", 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046225939999999998 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493109+0000", 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8202999999999995e-05 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:29.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290108+0000", 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79699814300000005 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290237+0000", 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012928200000000001 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290985+0000", 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000748064 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.677 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:29.678 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:29.678 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:29.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:29.680 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:29.677+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:29.757 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:29.757 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:29.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:29.780 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:29.778+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:29.800 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:29.800 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:29.800 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:29.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:29.880 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:29.878+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:29.918 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:29.918 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:29.918 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:29.918 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:29.918 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:29.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:29.981 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:29.979+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:30.036 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:238 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.112286441000002, 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011165420000000001, 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 238 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921316+0000", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3480000000000001e-06 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921322+0000", 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:02:30.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921333+0000", 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0779e-05 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921339+0000", 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9699999999999996e-06 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921387+0000", 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8588000000000003e-05 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921462+0000", 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4530999999999995e-05 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922070+0000", 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060823699999999997 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923628+0000", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001558219 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932393+0000", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087645399999999995 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932480+0000", 2024-09-17T16:02:30.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7435000000000006e-05 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:239 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 5] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.112246861000003, 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011291409000000001, 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 239 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921355+0000", 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921358+0000", 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.568e-06 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921365+0000", 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8040000000000004e-06 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921371+0000", 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.536e-06 2024-09-17T16:02:30.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922590+0000", 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012196679999999999 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922675+0000", 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4715000000000005e-05 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923334+0000", 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065929300000000003 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925081+0000", 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001747024 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932583+0000", 2024-09-17T16:02:30.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075011599999999998 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932646+0000", 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3200999999999993e-05 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:240 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:30.049 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.112208651000003, 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011490029000000001, 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 240 2024-09-17T16:02:30.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7899999999999999e-07 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921396+0000", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.52e-06 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921402+0000", 2024-09-17T16:02:30.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309999999999997e-06 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921408+0000", 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5690000000000004e-06 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923661+0000", 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022530979999999998 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923874+0000", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021281499999999999 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924326+0000", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045228500000000002 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925665+0000", 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013387049999999999 2024-09-17T16:02:30.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932831+0000", 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071659510000000003 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932883+0000", 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1975999999999998e-05 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:272 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.107920751999998, 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046207150000000001, 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.056 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 272 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925682+0000", 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7060000000000001e-06 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925690+0000", 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3950000000000003e-06 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925706+0000", 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.611e-05 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925713+0000", 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9809999999999997e-06 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2625000000000003e-05 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925900+0000", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011498 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926711+0000", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081025800000000001 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929013+0000", 2024-09-17T16:02:30.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023021669999999999 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930254+0000", 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012415900000000001 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930301+0000", 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6903000000000001e-05 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:273 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 8] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.107871834000001, 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070115810000000002, 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.062 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 273 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925730+0000", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925733+0000", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3900000000000002e-06 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925740+0000", 2024-09-17T16:02:30.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6610000000000003e-06 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925752+0000", 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2038000000000001e-05 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927223+0000", 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014708670000000001 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927310+0000", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7535999999999997e-05 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927853+0000", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054293600000000005 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929016+0000", 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001162445 2024-09-17T16:02:30.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932681+0000", 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003665092 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932741+0000", 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0327e-05 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:274 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.107821452000003, 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011049606999999999, 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.068 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 274 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.84e-07 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.425e-06 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925790+0000", 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2100000000000001e-06 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925793+0000", 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3409999999999999e-06 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928169+0000", 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023761490000000001 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928252+0000", 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2699000000000002e-05 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928677+0000", 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042531099999999999 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929281+0000", 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060397100000000002 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936793+0000", 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00751164 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936829+0000", 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6477000000000003e-05 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.073 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/32 ECSubWrite(tid=53, reqid=client.4560.0:344, at_version=54'24, trim_to=0'0, roll_forward_to=52'22))", 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.691352381999998, 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036631329999999998, 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:30.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8099999999999998e-07 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342277+0000", 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7398000000000001e-05 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342817+0000", 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054081699999999997 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342835+0000", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342920+0000", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4445999999999998e-05 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342932+0000", 2024-09-17T16:02:30.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2283000000000001e-05 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345862+0000", 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002929577 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345912+0000", 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0451000000000002e-05 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.078 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 64/53 ECSubWrite(tid=41, reqid=osd.15.0:41, at_version=64'23, trim_to=0'0, roll_forward_to=62'21))", 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.695508399000001, 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003803233, 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338094+0000", 2024-09-17T16:02:30.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1699999999999997e-07 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338127+0000", 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3801999999999999e-05 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338724+0000", 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059661000000000002 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338737+0000", 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3168e-05 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338818+0000", 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0558999999999999e-05 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338830+0000", 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2686e-05 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341825+0000", 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029948940000000001 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341896+0000", 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0797000000000007e-05 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.083 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 64/53 ECSubWrite(tid=43, reqid=osd.15.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=64'23))", 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345481+0000", 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.688120152, 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032921920000000002, 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345481+0000", 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345481+0000", 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345482+0000", 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5799999999999998e-07 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345503+0000", 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1066000000000001e-05 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345621+0000", 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117441 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345632+0000", 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1586000000000001e-05 2024-09-17T16:02:30.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345731+0000", 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9084000000000002e-05 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345742+0000", 2024-09-17T16:02:30.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0693000000000001e-05 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348728+0000", 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029865719999999998 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348773+0000", 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4792000000000001e-05 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs6 64/32 ECSubWrite(tid=68, reqid=osd.0.0:68, at_version=64'32, trim_to=0'0, roll_forward_to=62'30))", 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345546+0000", 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.688054893, 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003253779, 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345546+0000", 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345546+0000", 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345547+0000", 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.67e-07 2024-09-17T16:02:30.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345568+0000", 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1892999999999999e-05 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345668+0000", 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9763e-05 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345675+0000", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9120000000000001e-06 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345771+0000", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5748000000000006e-05 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345783+0000", 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1667e-05 2024-09-17T16:02:30.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348762+0000", 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029794219999999998 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348800+0000", 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8207000000000002e-05 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.688029563000001, 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047088649999999996, 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3500000000000002e-07 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345597+0000", 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4491e-05 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346172+0000", 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057511800000000001 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346180+0000", 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3480000000000004e-06 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346420+0000", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023985200000000001 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346431+0000", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1372000000000001e-05 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350228+0000", 2024-09-17T16:02:30.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037965479999999998 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350281+0000", 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800999999999998e-05 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s5 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:30.098 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.688009019999999, 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040542870000000002, 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4600000000000001e-07 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345620+0000", 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7722999999999999e-05 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345697+0000", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7019000000000002e-05 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345707+0000", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345799+0000", 2024-09-17T16:02:30.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2009999999999995e-05 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345809+0000", 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0036e-05 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349602+0000", 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037928200000000001 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349647+0000", 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4552000000000003e-05 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s5 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.687978007000002, 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047402829999999996, 2024-09-17T16:02:30.103 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345624+0000", 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345653+0000", 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9376000000000001e-05 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345739+0000", 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5702e-05 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345745+0000", 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2929999999999997e-06 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346167+0000", 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042231199999999999 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346178+0000", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0627e-05 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350309+0000", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041311509999999996 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350364+0000", 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4466000000000003e-05 2024-09-17T16:02:30.107 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.13s2 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.687327047, 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039770489999999999, 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1400000000000001e-07 2024-09-17T16:02:30.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346300+0000", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5800999999999999e-05 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346608+0000", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030764399999999998 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346618+0000", 2024-09-17T16:02:30.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.747e-06 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346718+0000", 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010035900000000001 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3066e-05 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350161+0000", 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034297310000000001 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350251+0000", 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0487000000000003e-05 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.18s1 69/53 ECSubWrite(tid=216, reqid=client.4560.0:480, at_version=69'34, trim_to=0'0, roll_forward_to=68'33))", 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.346649+0000", 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.686952107, 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035708599999999999, 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.346649+0000", 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.346649+0000", 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.346649+0000", 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5100000000000001e-07 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.347600+0000", 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00095048400000000003 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.347643+0000", 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2919e-05 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.347654+0000", 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0981999999999999e-05 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.347772+0000", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117977 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.347786+0000", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4514e-05 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350171+0000", 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023847679999999998 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350220+0000", 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8865000000000001e-05 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.17s3 73/65 ECSubWrite(tid=172, reqid=client.4560.0:581, at_version=73'38, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.620062240999999, 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93455132600000002, 2024-09-17T16:02:30.117 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413540+0000", 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413561+0000", 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1708999999999998e-05 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347419+0000", 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.933857559 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347428+0000", 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9579999999999996e-06 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347484+0000", 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6483000000000001e-05 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347492+0000", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.348050+0000", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055774199999999996 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.348090+0000", 2024-09-17T16:02:30.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0238999999999999e-05 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:631 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.604926395, 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085792290000000007, 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.122 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 631 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428677+0000", 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4420000000000001e-06 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428687+0000", 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.577e-06 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428709+0000", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1781999999999999e-05 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428720+0000", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.094e-05 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428800+0000", 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0217999999999997e-05 2024-09-17T16:02:30.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428809+0000", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431022+0000", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022127649999999998 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431120+0000", 2024-09-17T16:02:30.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7845999999999998e-05 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432156+0000", 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001036896 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435767+0000", 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003610753 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.437165+0000", 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013976769999999999 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.437254+0000", 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9155999999999998e-05 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:632 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 6] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:30.128 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.604854948, 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.009419696, 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 632 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428747+0000", 2024-09-17T16:02:30.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.68e-07 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428750+0000", 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1810000000000001e-06 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428760+0000", 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0198e-05 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428766+0000", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428881+0000", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011492 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428894+0000", 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3129000000000001e-05 2024-09-17T16:02:30.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432631+0000", 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003736971 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432729+0000", 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7295999999999998e-05 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433311+0000", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058199599999999999 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435769+0000", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024582969999999999 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438084+0000", 2024-09-17T16:02:30.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023153480000000001 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438166+0000", 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1771e-05 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:633 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:30.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.604806496, 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010126513, 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 633 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428796+0000", 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0599999999999999e-07 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428800+0000", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6330000000000004e-06 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428813+0000", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2906e-05 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428821+0000", 2024-09-17T16:02:30.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5619999999999998e-06 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428921+0000", 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010017 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428927+0000", 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9889999999999997e-06 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433618+0000", 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046906439999999999 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433715+0000", 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7182000000000006e-05 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434216+0000", 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050111399999999999 2024-09-17T16:02:30.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436519+0000", 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023034869999999999 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438845+0000", 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023260189999999999 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438921+0000", 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5901000000000004e-05 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 81/32 ECSubWrite(tid=102, reqid=osd.0.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=81'49))", 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.586743478000001, 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036110410000000002, 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.142 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446859+0000", 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4799999999999997e-07 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446872+0000", 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3324e-05 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446904+0000", 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2388000000000002e-05 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446913+0000", 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4910000000000005e-06 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447549+0000", 2024-09-17T16:02:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063597399999999998 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447560+0000", 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1453e-05 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.450429+0000", 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028682650000000001 2024-09-17T16:02:30.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.450469+0000", 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0297999999999999e-05 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.147 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.149 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:30.146+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:30.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:30.251 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:30.248+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:30.298 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.298 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:30.298 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:30.298 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:30.391 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:30.389+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:30.453 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:30.451+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:30.453 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:30.454 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:30.454 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:30.498 INFO:tasks.rados.rados.0.smithi181.stdout:326: done (0 left) 2024-09-17T16:02:30.498 INFO:tasks.rados.rados.0.smithi181.stdout:327: snap_remove snap 28 2024-09-17T16:02:30.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:02:30.553 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:30.551+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:30.611 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.611 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.680311635000002, 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:02:30.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.614 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.616 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=25, reqid=client.4560.0:264, at_version=45'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.689642644000003, 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026452619999999998, 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919303+0000", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.14e-07 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.919991+0000", 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00068760399999999997 2024-09-17T16:02:30.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920025+0000", 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4292000000000003e-05 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920043+0000", 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8028000000000001e-05 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920118+0000", 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4583000000000001e-05 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.920129+0000", 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1105e-05 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921921+0000", 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017915590000000001 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.921949+0000", 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7977e-05 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 45.683559819000003, 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.621 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:02:30.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.676734203000002, 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:02:30.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.630 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.257409795000001, 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017347922000000002, 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351537+0000", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351558+0000", 2024-09-17T16:02:30.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1399e-05 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367847+0000", 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289428000000002 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367858+0000", 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0835e-05 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367938+0000", 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0104000000000004e-05 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367950+0000", 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1153e-05 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368846+0000", 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089654899999999996 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368884+0000", 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8075999999999999e-05 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.215815687000003, 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047653560000000001, 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.635 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393131+0000", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9600000000000008e-07 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394022+0000", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089089400000000004 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394066+0000", 2024-09-17T16:02:30.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4193000000000002e-05 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394089+0000", 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2608000000000001e-05 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394165+0000", 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5638999999999999e-05 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394175+0000", 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0277e-05 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397846+0000", 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036707889999999998 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397896+0000", 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9960000000000003e-05 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.639 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=64'34, trim_to=0'0, roll_forward_to=62'32))", 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.271665261999999, 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035966639999999999, 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:30.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.04e-07 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337334+0000", 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3352e-05 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337390+0000", 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5319999999999999e-05 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337400+0000", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337509+0000", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108571 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337523+0000", 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4211e-05 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340822+0000", 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032988689999999998 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340877+0000", 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5481e-05 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.269655491999998, 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027044159999999999, 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339291+0000", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339292+0000", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.187e-06 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339317+0000", 2024-09-17T16:02:30.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5355e-05 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339357+0000", 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0244000000000003e-05 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339368+0000", 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0512e-05 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339491+0000", 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012284700000000001 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.339502+0000", 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0848e-05 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341950+0000", 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002448703 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341995+0000", 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4719999999999999e-05 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=42, reqid=osd.15.0:40, at_version=0'0, trim_to=0'0, roll_forward_to=64'34))", 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.263885008999999, 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027339, 2024-09-17T16:02:30.649 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345061+0000", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.28e-07 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345069+0000", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6920000000000002e-06 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345381+0000", 2024-09-17T16:02:30.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031179599999999998 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345390+0000", 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5810000000000003e-06 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345492+0000", 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101902 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345506+0000", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3395e-05 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347746+0000", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002240845 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347795+0000", 2024-09-17T16:02:30.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8461000000000002e-05 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s2 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.263633536, 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035907539999999998, 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345313+0000", 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:02:30.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345362+0000", 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8804000000000002e-05 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345410+0000", 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8133999999999997e-05 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345419+0000", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6039999999999999e-06 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345499+0000", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0173000000000006e-05 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345510+0000", 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348833+0000", 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033223749999999998 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348903+0000", 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0499000000000004e-05 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.658 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.262236251000001, 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034797959999999998, 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346711+0000", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6600000000000005e-07 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346735+0000", 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4652e-05 2024-09-17T16:02:30.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346773+0000", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7855999999999998e-05 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346782+0000", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5599999999999994e-06 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346854+0000", 2024-09-17T16:02:30.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2512e-05 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346865+0000", 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0957e-05 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350146+0000", 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032811709999999998 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350190+0000", 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3222000000000001e-05 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 76/32 ECSubWrite(tid=207, reqid=client.4560.0:597, at_version=76'132, trim_to=0'0, roll_forward_to=75'130))", 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 16.199800489000001, 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045991310000000002, 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.663 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410426+0000", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012808170000000001 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410463+0000", 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6449000000000003e-05 2024-09-17T16:02:30.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410471+0000", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8420000000000005e-06 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410556+0000", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5426999999999994e-05 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410574+0000", 2024-09-17T16:02:30.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7306999999999999e-05 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413700+0000", 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031262460000000001 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413745+0000", 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4910000000000002e-05 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.176977791000001, 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036738819999999998, 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431969+0000", 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3500000000000002e-07 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431996+0000", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7461e-05 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432037+0000", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0552000000000001e-05 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432059+0000", 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2345e-05 2024-09-17T16:02:30.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432134+0000", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5159000000000003e-05 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432145+0000", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0875e-05 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435607+0000", 2024-09-17T16:02:30.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034618510000000002 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435642+0000", 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5203999999999997e-05 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=225, reqid=client.4560.0:627, at_version=79'152, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.175931549, 2024-09-17T16:02:30.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003614744, 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:30.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433015+0000", 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433165+0000", 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149612 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433313+0000", 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014852600000000001 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433322+0000", 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4770000000000003e-06 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433429+0000", 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010759099999999999 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433441+0000", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1323e-05 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436598+0000", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031569749999999998 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436629+0000", 2024-09-17T16:02:30.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1473000000000002e-05 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 13.175774562000001, 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034963870000000001, 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433181+0000", 2024-09-17T16:02:30.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7489999999999993e-06 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433276+0000", 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4340999999999997e-05 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433357+0000", 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1524000000000001e-05 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433364+0000", 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5710000000000003e-06 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434090+0000", 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072590199999999995 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434157+0000", 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7516999999999997e-05 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436645+0000", 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024882789999999999 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436668+0000", 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2504e-05 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 80/32 ECSubWrite(tid=233, reqid=client.4560.0:664, at_version=80'156, trim_to=0'0, roll_forward_to=79'154))", 2024-09-17T16:02:30.681 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.162082469, 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0030183900000000001, 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.446864+0000", 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1899999999999999e-07 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447421+0000", 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055686899999999998 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447452+0000", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1937999999999998e-05 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447462+0000", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3090000000000001e-06 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447533+0000", 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1507999999999994e-05 2024-09-17T16:02:30.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.447546+0000", 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3174e-05 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.449848+0000", 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002301403 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:18.449882+0000", 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4069999999999997e-05 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=240, reqid=osd.12.0:240, at_version=81'161, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.179416874999999, 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003961633, 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.686 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:30.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429579+0000", 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9345999999999998e-05 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430294+0000", 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071481400000000003 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430305+0000", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1618e-05 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430487+0000", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018190799999999999 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430503+0000", 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.643e-05 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433445+0000", 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029419300000000002 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433491+0000", 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5348e-05 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=241, reqid=osd.12.0:241, at_version=81'163, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.179363649000001, 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0053980119999999998, 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429583+0000", 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2399999999999999e-07 2024-09-17T16:02:30.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429601+0000", 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8195000000000001e-05 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430343+0000", 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074222800000000005 2024-09-17T16:02:30.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430351+0000", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5329999999999997e-06 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431146+0000", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079573700000000003 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431157+0000", 2024-09-17T16:02:30.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0696999999999999e-05 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434934+0000", 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037763610000000002 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434980+0000", 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6836999999999999e-05 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.177640870999999, 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036787539999999998, 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.29e-07 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431329+0000", 2024-09-17T16:02:30.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3555000000000001e-05 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431411+0000", 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1885999999999995e-05 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431421+0000", 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0139999999999999e-05 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431508+0000", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7016999999999997e-05 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431520+0000", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2160999999999999e-05 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434938+0000", 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034178939999999999 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434984+0000", 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5871999999999997e-05 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:30.700 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.177614401, 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040176439999999999, 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:30.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431352+0000", 2024-09-17T16:02:30.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0449e-05 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431581+0000", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022874899999999999 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431589+0000", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0299999999999994e-06 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.432989+0000", 2024-09-17T16:02:30.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014002800000000001 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433001+0000", 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2004999999999999e-05 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435321+0000", 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002319883 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:30.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435349+0000", 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8008e-05 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:30.705 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:30.707 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:30.703+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:30.807 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:30.805+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:30.814 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.814 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:30.814 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:30.814 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:30.909 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:30.907+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:30.970 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:30.970 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:30.970 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:30.970 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:30.970 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:30.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:31.022 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:31.020+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 34/32 ECSubWrite(tid=2, reqid=client.4560.0:10, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:31.087 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.227595977, 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019348939999999999, 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856949+0000", 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1200000000000001e-07 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.856995+0000", 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5192999999999999e-05 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857072+0000", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7515999999999994e-05 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857095+0000", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2942000000000002e-05 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857191+0000", 2024-09-17T16:02:31.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6087000000000003e-05 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.857207+0000", 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6256000000000002e-05 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858835+0000", 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001627133 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.858883+0000", 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8854999999999999e-05 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.224633484999998, 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020251290000000001, 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.092 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859911+0000", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.859912+0000", 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860021+0000", 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109295 2024-09-17T16:02:31.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860069+0000", 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7791999999999999e-05 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860083+0000", 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3698e-05 2024-09-17T16:02:31.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860157+0000", 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4707000000000001e-05 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.860167+0000", 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4120000000000007e-06 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861892+0000", 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001725465 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.861936+0000", 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3968999999999999e-05 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.098 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.221003154999998, 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020318110000000001, 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863541+0000", 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863542+0000", 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3300000000000002e-07 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863604+0000", 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1809000000000004e-05 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863694+0000", 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0345000000000003e-05 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.863701+0000", 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9909999999999996e-06 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864177+0000", 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000475699 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.864186+0000", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4630000000000007e-06 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865542+0000", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013557339999999999 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865573+0000", 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1236999999999999e-05 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.103 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.196842330000003, 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:02:31.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.192290370000002, 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:02:31.109 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:02:31.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.113 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.4s5 45/32 ECSubWrite(tid=11, reqid=client.4560.0:272, at_version=45'10, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.157887430999999, 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019463340000000001, 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926657+0000", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9200000000000002e-07 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926680+0000", 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3037e-05 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926741+0000", 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0134999999999999e-05 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926751+0000", 2024-09-17T16:02:31.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0315e-05 2024-09-17T16:02:31.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926822+0000", 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0667000000000006e-05 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926854+0000", 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2320999999999998e-05 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928578+0000", 2024-09-17T16:02:31.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017237979999999999 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928603+0000", 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5769e-05 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.119 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.166111661000002, 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020717940000000001, 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.120 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.121 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918433+0000", 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6899999999999999e-07 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918457+0000", 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.122 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918505+0000", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7648000000000002e-05 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918514+0000", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8609999999999993e-06 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918721+0000", 2024-09-17T16:02:31.123 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020663600000000001 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.918732+0000", 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1058e-05 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920477+0000", 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017447840000000001 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.124 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.920505+0000", 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7815000000000001e-05 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 46/32 ECSubWrite(tid=36, reqid=osd.0.0:36, at_version=46'14, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:31.125 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.160275757999997, 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020194549999999999, 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.126 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924269+0000", 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1600000000000002e-07 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924293+0000", 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3822999999999999e-05 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.127 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924390+0000", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6893999999999998e-05 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924399+0000", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7919999999999998e-06 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924465+0000", 2024-09-17T16:02:31.128 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309000000000005e-05 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.924474+0000", 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5289999999999997e-06 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926235+0000", 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001760529 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.129 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.926288+0000", 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3263000000000001e-05 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s6 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.711543304000003, 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003094585, 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.130 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.131 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.61e-07 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373230+0000", 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022871100000000001 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373269+0000", 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8470999999999998e-05 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.132 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373278+0000", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8449999999999998e-06 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373359+0000", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0401999999999994e-05 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373368+0000", 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1919999999999994e-06 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.133 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376052+0000", 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026842319999999999 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376096+0000", 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3571e-05 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.134 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.687718320000002, 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020885859999999999, 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.135 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397062+0000", 2024-09-17T16:02:31.136 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236006 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397106+0000", 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3995000000000001e-05 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397119+0000", 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.137 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397201+0000", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2498999999999997e-05 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397214+0000", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2819999999999999e-05 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398870+0000", 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016554829999999999 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.138 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398915+0000", 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5083000000000002e-05 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.741874395, 2024-09-17T16:02:31.139 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038094829999999998, 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.140 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342671+0000", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342695+0000", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4366999999999999e-05 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343531+0000", 2024-09-17T16:02:31.141 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083620199999999997 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343542+0000", 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0672e-05 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343828+0000", 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000285834 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.142 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343846+0000", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8471999999999999e-05 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346440+0000", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025931499999999998 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346480+0000", 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0064000000000001e-05 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.143 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.738804273, 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027362839999999999, 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.144 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1799999999999999e-07 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.145 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0065999999999999e-05 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345845+0000", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4876999999999996e-05 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345856+0000", 2024-09-17T16:02:31.146 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1073000000000001e-05 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345935+0000", 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9010000000000004e-05 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345947+0000", 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1708999999999999e-05 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.147 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348437+0000", 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024899560000000002 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348477+0000", 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9375000000000002e-05 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.148 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s5 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.738763648999999, 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034936559999999999, 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.149 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.92e-07 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345815+0000", 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3519999999999998e-05 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.150 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345868+0000", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3248000000000002e-05 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345877+0000", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7970000000000006e-06 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346334+0000", 2024-09-17T16:02:31.151 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045731999999999999 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346345+0000", 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0854e-05 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349226+0000", 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028807569999999998 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.152 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349275+0000", 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8968000000000003e-05 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.737912377000001, 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002590527, 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.153 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.154 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.73e-07 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346659+0000", 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7055000000000001e-05 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346738+0000", 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8388999999999998e-05 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.155 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346746+0000", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4090000000000003e-06 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346817+0000", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0895999999999993e-05 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346826+0000", 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-17T16:02:31.156 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349183+0000", 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023570639999999999 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349223+0000", 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607999999999998e-05 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.157 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 28.73788265, 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037577729999999998, 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.158 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.159 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346683+0000", 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0866999999999999e-05 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347481+0000", 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079794800000000004 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347491+0000", 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.160 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348761+0000", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001270188 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348771+0000", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.815e-06 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350377+0000", 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016056499999999999 2024-09-17T16:02:31.161 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350420+0000", 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2944000000000001e-05 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:31.162 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.733580216, 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027021179999999999, 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.163 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350965+0000", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.99e-07 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350989+0000", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4145999999999999e-05 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.164 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351039+0000", 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9858000000000003e-05 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351051+0000", 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2729e-05 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351123+0000", 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1680000000000005e-05 2024-09-17T16:02:31.165 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351135+0000", 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2167000000000001e-05 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353616+0000", 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002481287 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.166 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353666+0000", 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0052000000000001e-05 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.638990701999999, 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026861049999999998, 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.167 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.168 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445555+0000", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9299999999999997e-07 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446015+0000", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046028900000000001 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446034+0000", 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9279e-05 2024-09-17T16:02:31.169 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446041+0000", 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446100+0000", 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8473000000000001e-05 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.170 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446107+0000", 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9319999999999999e-06 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448192+0000", 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020857570000000001 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448240+0000", 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7469000000000002e-05 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.171 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 81/53 ECSubWrite(tid=47, reqid=osd.8.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=81'22))", 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.637664990999999, 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002323982, 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:31.172 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446881+0000", 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.232e-06 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.173 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446896+0000", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4976e-05 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446962+0000", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6397999999999997e-05 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446971+0000", 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2609999999999999e-06 2024-09-17T16:02:31.174 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447062+0000", 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0257e-05 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447074+0000", 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2108e-05 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.175 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449155+0000", 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020816559999999999 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449204+0000", 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8093999999999997e-05 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.176 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 81/32 ECSubWrite(tid=102, reqid=osd.0.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=81'49))", 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.63761371, 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022743210000000002, 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.177 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446942+0000", 2024-09-17T16:02:31.178 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0988999999999999e-05 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447265+0000", 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032216900000000003 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447274+0000", 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4800000000000007e-06 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.179 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447363+0000", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9007999999999999e-05 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447374+0000", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1121000000000001e-05 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449154+0000", 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001779997 2024-09-17T16:02:31.180 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449205+0000", 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0948000000000001e-05 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:31.181 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 11.635692586999999, 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023273109999999999, 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:31.182 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448853+0000", 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449276+0000", 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000422757 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.183 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449299+0000", 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3098000000000001e-05 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449308+0000", 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0159999999999997e-06 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.184 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449374+0000", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6029000000000001e-05 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449385+0000", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1019000000000001e-05 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451144+0000", 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001759287 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.185 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4966999999999999e-05 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.186 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:31.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:31.188 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:31.185+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:31.289 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:31.287+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:31.335 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:31.335 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:31.335 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:31.335 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:31.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:31.389 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:31.388+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:31.451 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:31.451 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:31.451 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:31.451 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:31.451 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:31.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:31.490 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:31.489+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:31.496 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:02:31.501 INFO:tasks.rados.rados.0.smithi181.stdout:327: done (0 left) 2024-09-17T16:02:31.501 INFO:tasks.rados.rados.0.smithi181.stdout:328: snap_create 2024-09-17T16:02:31.577 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.721639201999999, 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:02:31.578 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:31.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:02:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:31.584 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:02:31.584 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:02:31.584 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.584 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.584 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:31.586 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.708961907999999, 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:02:31.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:02:31.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:02:31.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.593 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.708688666999997, 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.594 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:02:31.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:31.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:02:31.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.705822885000003, 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:02:31.600 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.705770958000002, 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.606 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:31.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:02:31.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.613 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.685230191999999, 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.615 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:02:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:02:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.685147018000002, 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:02:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.670704250999997, 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:02:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:02:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:02:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.670664561000002, 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:02:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:02:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.635422321, 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:388 3.16s0 3:6d55c691:::smithi18136759-16:c [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.236251473000003, 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.148899007, 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 388 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338147+0000", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.129e-06 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338155+0000", 2024-09-17T16:02:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1389999999999998e-06 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338160+0000", 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338288+0000", 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128374 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338295+0000", 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7730000000000002e-06 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.342955+0000", 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046598580000000002 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343006+0000", 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1329000000000003e-05 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487043+0000", 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14403680199999999 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.16s0 60/53 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.230820764000001, 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14348618299999999, 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343575+0000", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6400000000000003e-07 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343629+0000", 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4472000000000003e-05 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486547+0000", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.142917403 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486554+0000", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.503e-06 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486609+0000", 2024-09-17T16:02:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4527000000000002e-05 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487061+0000", 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045181399999999998 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "replica_scrub(pg: 3.15s1,from:0'0,to:56'8,epoch:60/53,start:3:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.087876825000002, 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45217676400000001, 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486519+0000", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486525+0000", 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5949999999999998e-06 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938489+0000", 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45196369600000003 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938505+0000", 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6064000000000001e-05 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938656+0000", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015132599999999999 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938666+0000", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6320000000000003e-06 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938695+0000", 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9652e-05 2024-09-17T16:02:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:447 3.1s0 3:8347559a:::smithi18136759-7:10 [read 97754~16934,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.233527942999999, 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014398077, 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 447 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340869+0000", 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340875+0000", 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6630000000000004e-06 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340887+0000", 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.112e-05 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340896+0000", 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344260+0000", 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033644320000000001 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344265+0000", 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2479999999999998e-06 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353223+0000", 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089583140000000002 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353351+0000", 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128231 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355265+0000", 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019143319999999999 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:448 3.1s0 3:8347559a:::smithi18136759-7:10 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.233472336999998, 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015642311999999998, 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 448 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340927+0000", 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.569e-06 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340936+0000", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340942+0000", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344342+0000", 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034007899999999999 2024-09-17T16:02:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344344+0000", 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.333e-06 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355309+0000", 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010965557000000001 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355379+0000", 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9503000000000002e-05 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356565+0000", 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011865269999999999 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.667 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:449 3.1s0 3:8347559a:::smithi18136759-7:10 [read 41111~73577,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.233443393000002, 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015941349, 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.668 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 449 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4400000000000001e-07 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340955+0000", 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0299999999999998e-06 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340962+0000", 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.17e-06 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340968+0000", 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344372+0000", 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034039690000000002 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344373+0000", 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1400000000000001e-06 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355572+0000", 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011199565 2024-09-17T16:02:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355632+0000", 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9210999999999999e-05 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356893+0000", 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012616330000000001 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:511 3.6s0 3:60a7d8b5:::smithi18136759-22:head [copy-from ver 12] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 23.199976832000001, 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016414522000000001, 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 511 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374421+0000", 2024-09-17T16:02:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2189999999999998e-06 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374429+0000", 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4929999999999998e-06 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374449+0000", 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0360000000000002e-05 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374459+0000", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0559999999999994e-06 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374533+0000", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4171999999999994e-05 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374651+0000", 2024-09-17T16:02:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118495 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.383140+0000", 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084888240000000007 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.384289+0000", 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011493639999999999 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390716+0000", 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064265609999999999 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390833+0000", 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116978 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/53 ECSubWrite(tid=138, reqid=osd.14.0:138, at_version=74'39, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.170459530999999, 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.11895731, 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:02:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403952+0000", 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5957e-05 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522019+0000", 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.118066429 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522040+0000", 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1769000000000001e-05 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522199+0000", 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015867999999999999 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522215+0000", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5971999999999999e-05 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522867+0000", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065141500000000002 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522893+0000", 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6505000000000001e-05 2024-09-17T16:02:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:772 3.16s0 3:6d55c691:::smithi18136759-16:head [copy-from ver 39] snapc 21=[21,20,1f,1e,1d,1c,19,18,16,10,c,6] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.1020049649999999, 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015349042, 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 772 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472392+0000", 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7349999999999999e-06 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472399+0000", 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0469999999999996e-06 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472416+0000", 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7073999999999999e-05 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472424+0000", 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3580000000000003e-06 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472510+0000", 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5284000000000005e-05 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472516+0000", 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.474786+0000", 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022700929999999999 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.474869+0000", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2621999999999996e-05 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.484369+0000", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095008660000000002 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.485566+0000", 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011963360000000001 2024-09-17T16:02:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.487506+0000", 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001940322 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.487739+0000", 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000233284 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 87/65 ECSubWriteReply(tid=220, last_complete=87'36, committed=1, applied=1))", 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.0856082909999998, 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54478348899999995, 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6300000000000001e-07 2024-09-17T16:02:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488789+0000", 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.567e-06 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033437+0000", 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54464779200000002 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033444+0000", 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9099999999999999e-06 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033516+0000", 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2305999999999995e-05 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033570+0000", 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4551e-05 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:31.696 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:31.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:31.698 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:31.695+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:31.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:31.798 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:31.795+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:31.812 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:31.813 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:31.813 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:31.813 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:31.821 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:31.899 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:31.896+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:31.902 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:31.902 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:31.925 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:31.925 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:31.946 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:31.964 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:31.965 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:31.967 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:31.968 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:31.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:31.988 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:31.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:32.010 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:32.010 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:32.017 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:32.015+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:32.031 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:32.031 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:32.031 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:32.052 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:32.052 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:32.073 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:32.073 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:32.077 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.077 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.086 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:32.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:32.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:32.116 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:32.116 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:32.127 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:32.125+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:32.231 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:32.231 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:32.231 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:32.228+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:32.232 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.232 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:32.232 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:32.233 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:32.233 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:32.239 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:32.331 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:32.329+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:32.384 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.384 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.393 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:32.433 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:32.430+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:32.501 INFO:tasks.rados.rados.0.smithi181.stdout:328: done (0 left) 2024-09-17T16:02:32.501 INFO:tasks.rados.rados.0.smithi181.stdout:329: snap_create 2024-09-17T16:02:32.533 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:32.531+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:32.538 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.538 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.539 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:32.539 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:32.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:32.547 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:32.634 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:32.632+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:32.692 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.693 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.701 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:32.734 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:32.733+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:32.803 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.803 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.811 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:32.836 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:32.834+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:32.914 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:32.914 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:32.923 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:32.936 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:32.935+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:33.026 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.026 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.035 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:33.036 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:33.034+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:33.137 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:33.135+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:33.140 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.140 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:33.238 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:33.236+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:33.249 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.249 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.258 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:33.339 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:33.337+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:33.373 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.373 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.382 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:33.440 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:33.438+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:33.487 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.487 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:33.503 INFO:tasks.rados.rados.0.smithi181.stdout:329: done (0 left) 2024-09-17T16:02:33.503 INFO:tasks.rados.rados.0.smithi181.stdout:330: snap_remove snap 37 2024-09-17T16:02:33.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:02:33.541 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:33.539+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:33.606 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.606 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:33.642 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:33.639+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:33.718 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.718 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:33.743 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:33.740+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:33.829 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.829 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:33.844 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:33.842+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:33.939 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:33.940 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:33.945 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:33.943+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:33.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:34.046 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:34.044+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:34.051 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:34.051 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:34.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:34.147 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:34.145+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:34.162 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:34.162 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:34.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:34.255 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:34.253+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:34.313 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:34.313 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:34.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:34.367 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:34.364+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:34.435 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:34.435 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:34.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:34.454 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:34.452+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:34.501 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:02:34.504 INFO:tasks.rados.rados.0.smithi181.stdout:330: done (0 left) 2024-09-17T16:02:34.504 INFO:tasks.rados.rados.0.smithi181.stdout:331: snap_remove snap 33 2024-09-17T16:02:34.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:34.508 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:34.509 INFO:tasks.thrashosds.thrasher:Reweighting osd 10 to 0.7851943748398196 2024-09-17T16:02:34.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 0.7851943748398196 2024-09-17T16:02:34.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:34.554 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:34.552+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:34.599 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:34.600 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:34.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:34.655 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:34.653+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:34.713 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:34.714 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:34.752 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:34.752 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:34.756 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:34.754+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:34.789 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:34.789 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:34.799 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:248 3.cs0 3:3808462d:::smithi18136759-6:head [copy-from ver 2] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 51.868218831, 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019268496999999999, 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 248 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928583+0000", 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928588+0000", 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2220000000000001e-06 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928601+0000", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2717e-05 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928608+0000", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8600000000000004e-06 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928671+0000", 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3791e-05 2024-09-17T16:02:34.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928756+0000", 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4293000000000001e-05 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940817+0000", 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01206144 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.942933+0000", 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021155280000000002 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947791+0000", 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048580639999999996 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947850+0000", 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9682999999999998e-05 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.805 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:249 3.cs0 3:3808462d:::smithi18136759-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 51.868177719000002, 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013329483, 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.806 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928624+0000", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2699999999999999e-07 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928626+0000", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8940000000000002e-06 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928643+0000", 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6988999999999999e-05 2024-09-17T16:02:34.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928651+0000", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.267e-06 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928901+0000", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025058200000000001 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928955+0000", 2024-09-17T16:02:34.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3973e-05 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941770+0000", 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012814828 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941852+0000", 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083000000000007e-05 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941953+0000", 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010034 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:285 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 48.864995399999998, 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010247877000000001, 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.811 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 285 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931808+0000", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8779999999999998e-06 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931822+0000", 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3828e-05 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931850+0000", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7500000000000001e-05 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931863+0000", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3738e-05 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931969+0000", 2024-09-17T16:02:34.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001052 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931976+0000", 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938082+0000", 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061057769999999997 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938176+0000", 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3714000000000004e-05 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939069+0000", 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00089279400000000003 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941333+0000", 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022647380000000001 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941992+0000", 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065853199999999998 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.942053+0000", 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1447999999999998e-05 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:286 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 2] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 48.86484171, 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013358431, 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.818 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 286 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931960+0000", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931965+0000", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9010000000000002e-06 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931974+0000", 2024-09-17T16:02:34.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2520000000000005e-06 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931981+0000", 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5950000000000004e-06 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932071+0000", 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0390999999999996e-05 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:34.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932079+0000", 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2679999999999994e-06 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939566+0000", 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074865030000000003 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939653+0000", 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7336999999999994e-05 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940238+0000", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058448400000000002 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944090+0000", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00385192 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945232+0000", 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001142784 2024-09-17T16:02:34.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945317+0000", 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5080999999999994e-05 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:287 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 48.864805771, 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013571456000000001, 2024-09-17T16:02:34.824 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 287 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931996+0000", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931999+0000", 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4300000000000002e-06 2024-09-17T16:02:34.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932006+0000", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2729999999999997e-06 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932012+0000", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8089999999999998e-06 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932107+0000", 2024-09-17T16:02:34.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4751000000000001e-05 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932112+0000", 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940481+0000", 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083682089999999997 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940546+0000", 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5704000000000007e-05 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940923+0000", 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00037686099999999998 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944112+0000", 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031884999999999999 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945503+0000", 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001391226 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945566+0000", 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3510999999999996e-05 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.830 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 52/32 ECSubWriteReply(tid=51, last_complete=52'22, committed=1, applied=1))", 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 44.439829617999997, 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.138524809, 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356972+0000", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356976+0000", 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2950000000000004e-06 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495052+0000", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13807575 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495065+0000", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3276e-05 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495133+0000", 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8274000000000004e-05 2024-09-17T16:02:34.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495496+0000", 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036245899999999998 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:344 3.fs0 3:fd9ed431:::smithi18136759-28:head [stat,delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.460837689000002, 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012105121, 2024-09-17T16:02:34.834 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 344 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335966+0000", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.004e-06 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335978+0000", 2024-09-17T16:02:34.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1810999999999999e-05 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336003+0000", 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5344e-05 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336017+0000", 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3901e-05 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.837 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336125+0000", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107707 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336136+0000", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1476e-05 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340410+0000", 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004273453 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340525+0000", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115017 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.341450+0000", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00092533800000000005 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.345523+0000", 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040730929999999999 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348008+0000", 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024844609999999999 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348068+0000", 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0516000000000001e-05 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=128, reqid=osd.12.0:128, at_version=64'68, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.458902518000002, 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076656809999999997, 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337917+0000", 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8366999999999999e-05 2024-09-17T16:02:34.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339622+0000", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017049529999999999 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339629+0000", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3660000000000003e-06 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339905+0000", 2024-09-17T16:02:34.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027553099999999997 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339915+0000", 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0124e-05 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345540+0000", 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056253780000000003 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799e-05 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.457144032000002, 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081855320000000006, 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.845 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:34.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339692+0000", 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4459000000000003e-05 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339726+0000", 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4029000000000002e-05 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339733+0000", 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0790000000000003e-06 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340271+0000", 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00053823 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340282+0000", 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347801+0000", 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075190999999999999 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347842+0000", 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1387999999999997e-05 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.849 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.10s6 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.451237140000003, 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011676772, 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.29e-07 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8580000000000001e-05 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346013+0000", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00042032499999999998 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346021+0000", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2269999999999993e-06 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346353+0000", 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033160900000000002 2024-09-17T16:02:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346363+0000", 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0390999999999999e-05 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357206+0000", 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010843148 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357240+0000", 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4263e-05 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.450330086999998, 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010663848, 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.854 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7099999999999998e-07 2024-09-17T16:02:34.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346489+0000", 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7731999999999999e-05 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346664+0000", 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017518199999999999 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346671+0000", 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1239999999999999e-06 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346721+0000", 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.02e-05 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.55e-06 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357095+0000", 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010364226000000001 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357135+0000", 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9563000000000001e-05 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s5 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:34.858 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.448950410999998, 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092773109999999999, 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347851+0000", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347861+0000", 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0325e-05 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347893+0000", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1173000000000001e-05 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347900+0000", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.306e-06 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347964+0000", 2024-09-17T16:02:34.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3678999999999998e-05 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347973+0000", 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0790000000000007e-06 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357093+0000", 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091199190000000006 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357128+0000", 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5055999999999998e-05 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.447674671000001, 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090524279999999995, 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.863 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349127+0000", 2024-09-17T16:02:34.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349137+0000", 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349186+0000", 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8306999999999997e-05 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349195+0000", 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3640000000000005e-06 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349414+0000", 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021941500000000001 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349424+0000", 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358144+0000", 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087197060000000007 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358179+0000", 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4814999999999998e-05 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.867 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=132, reqid=osd.12.0:132, at_version=64'72, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.445219760000001, 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076285509999999999, 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351582+0000", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351615+0000", 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2962000000000001e-05 2024-09-17T16:02:34.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351654+0000", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8847999999999998e-05 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351662+0000", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0099999999999995e-06 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351779+0000", 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117569 2024-09-17T16:02:34.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351788+0000", 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5639999999999997e-06 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359157+0000", 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073690580000000004 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359210+0000", 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2769999999999998e-05 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 32.444354277999999, 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079374270000000004, 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.872 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352447+0000", 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352467+0000", 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0146999999999999e-05 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352504+0000", 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6365000000000002e-05 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352512+0000", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352565+0000", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3613999999999999e-05 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352576+0000", 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1141999999999999e-05 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360356+0000", 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007779584 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360384+0000", 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7847999999999998e-05 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.443379837999998, 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70314915600000005, 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.54e-07 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353470+0000", 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9177e-05 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055585+0000", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70211433599999995 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055600+0000", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4989e-05 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055790+0000", 2024-09-17T16:02:34.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019022599999999999 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055803+0000", 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3455000000000001e-05 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056540+0000", 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000736289 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056570+0000", 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0530000000000001e-05 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:490 3.fs0 3:fd9ed431:::smithi18136759-28:head [append 0~770048] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.427782099000002, 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007518299, 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:34.881 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 490 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369020+0000", 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.322e-06 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369811+0000", 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00079078099999999995 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369823+0000", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.252e-05 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369829+0000", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4140000000000002e-06 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369879+0000", 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0158999999999999e-05 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369990+0000", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111549 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.370711+0000", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00072055199999999995 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.371929+0000", 2024-09-17T16:02:34.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001218217 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376334+0000", 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044050360000000002 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376537+0000", 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020274900000000001 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:494 3.fs0 3:fd9ed431:::smithi18136759-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.423131241, 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082749039999999996, 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 494 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.887 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.51e-07 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373672+0000", 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2689999999999998e-06 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373678+0000", 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2909999999999998e-06 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373681+0000", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3730000000000001e-06 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376047+0000", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002366071 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376106+0000", 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8610000000000003e-05 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381155+0000", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050492460000000003 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381250+0000", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5358000000000004e-05 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381944+0000", 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069423499999999997 2024-09-17T16:02:34.891 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 81/53 ECSubWrite(tid=74, reqid=osd.15.0:74, at_version=81'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.369508012000001, 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092459199999999995, 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6199999999999999e-07 2024-09-17T16:02:34.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427313+0000", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0389000000000001e-05 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427799+0000", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00048568899999999998 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427805+0000", 2024-09-17T16:02:34.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1040000000000003e-06 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429581+0000", 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017761179999999999 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429592+0000", 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0621e-05 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436494+0000", 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069020920000000003 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436539+0000", 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4545000000000001e-05 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s5 85/65 ECSubWrite(tid=214, reqid=client.4560.0:753, at_version=85'24, trim_to=0'0, roll_forward_to=84'20))", 2024-09-17T16:02:34.896 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.334529947, 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16302681799999999, 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:34.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462272+0000", 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462292+0000", 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0965999999999999e-05 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624643+0000", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.162350088 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624650+0000", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4290000000000004e-06 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624728+0000", 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7996000000000003e-05 2024-09-17T16:02:34.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624749+0000", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0990000000000001e-05 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625270+0000", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052091300000000002 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625298+0000", 2024-09-17T16:02:34.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7733999999999999e-05 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:34.901 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:34.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:34.903 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:34.903 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:34.903 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:34.904 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:34.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:34.905 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:34.905 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:34.905 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:34.905 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:34.906 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:34.906 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:34.900+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:34.924 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:34.924 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:34.946 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:34.946 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:35.003 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:35.001+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:35.014 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.014 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:35.014 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:35.014 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:35.092 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:35.092 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:35.104 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:35.102+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:35.113 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:35.113 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:35.135 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:35.135 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:35.156 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:35.156 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:35.162 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:35.159+0000 7f7a9ffff640 0 -- 172.21.15.6:0/754198114 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f7aa821fc30 msgr2=0x7f7aa8222020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:35.170 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.170 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:35.170 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:35.170 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:35.170 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:35.220 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:35.217+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:35.305 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:35.302+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:35.330 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.330 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:35.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:35.330 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:35.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=182, reqid=osd.11.0:182, at_version=46'20, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.409429744000001, 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066826129999999996, 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5800000000000001e-07 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917886+0000", 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6931999999999999e-05 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918192+0000", 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030561400000000001 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918355+0000", 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016328600000000001 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918431+0000", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5690000000000002e-05 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918441+0000", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9450000000000005e-06 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924513+0000", 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060725709999999997 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924542+0000", 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8317e-05 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.408750800999996, 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060611290000000002, 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.335 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.96e-07 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0068000000000001e-05 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010144 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918678+0000", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0900000000000005e-06 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918750+0000", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2100000000000004e-05 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918761+0000", 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0767999999999999e-05 2024-09-17T16:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924554+0000", 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057936560000000003 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924599+0000", 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4710999999999999e-05 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.339 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.408713701000003, 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059181870000000001, 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.340 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918576+0000", 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918591+0000", 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5583999999999999e-05 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918817+0000", 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022566799999999999 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918825+0000", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9389999999999997e-06 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918894+0000", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9332000000000006e-05 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918904+0000", 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9860000000000006e-06 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924448+0000", 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055442470000000004 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924493+0000", 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4947000000000002e-05 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.344 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.407741090999998, 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056807560000000003, 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919549+0000", 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919571+0000", 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2155999999999999e-05 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919754+0000", 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018332299999999999 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919767+0000", 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2656e-05 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920277+0000", 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051049799999999996 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920289+0000", 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1388e-05 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925184+0000", 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048955539999999999 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925228+0000", 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4202000000000001e-05 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.407701307000004, 2024-09-17T16:02:35.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056906200000000004, 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:35.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1299999999999999e-07 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919607+0000", 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9062000000000001e-05 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919796+0000", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018866 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919803+0000", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4150000000000002e-06 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920325+0000", 2024-09-17T16:02:35.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052165800000000004 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920334+0000", 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925242+0000", 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049072009999999999 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925278+0000", 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6634000000000002e-05 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.968527186999999, 2024-09-17T16:02:35.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067241940000000002, 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:35.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358763+0000", 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1710000000000001e-06 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358778+0000", 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5158e-05 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358817+0000", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8832999999999999e-05 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358826+0000", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2390000000000008e-06 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358909+0000", 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2556e-05 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358920+0000", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0910000000000001e-05 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365448+0000", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006528748 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365486+0000", 2024-09-17T16:02:35.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7579000000000001e-05 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/53 ECSubWrite(tid=23, reqid=client.4560.0:346, at_version=54'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.967792218, 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059831219999999996, 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.360 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359521+0000", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3705000000000001e-05 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359561+0000", 2024-09-17T16:02:35.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9879999999999998e-05 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359570+0000", 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1640000000000007e-06 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359645+0000", 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4765000000000007e-05 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359656+0000", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0702e-05 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365440+0000", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057839320000000003 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365480+0000", 2024-09-17T16:02:35.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0219000000000002e-05 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 69/32 ECSubWrite(tid=70, reqid=osd.2.0:70, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.988444868999999, 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24156233099999999, 2024-09-17T16:02:35.365 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:35.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338868+0000", 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3495000000000001e-05 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579761+0000", 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24089318000000001 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579766+0000", 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579834+0000", 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7265000000000001e-05 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579841+0000", 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.029e-06 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580381+0000", 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000540626 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580406+0000", 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4958999999999999e-05 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s5 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.973948303, 2024-09-17T16:02:35.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32307851999999998, 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353366+0000", 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5431000000000001e-05 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675724+0000", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32235754599999999 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675729+0000", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1100000000000002e-06 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675814+0000", 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.551e-05 2024-09-17T16:02:35.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675822+0000", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676394+0000", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057224999999999995 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676419+0000", 2024-09-17T16:02:35.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5049e-05 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.895238043999999, 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068757990000000001, 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.375 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:35.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432072+0000", 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0971e-05 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434134+0000", 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002062342 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434144+0000", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434203+0000", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8757e-05 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434214+0000", 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0879e-05 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438888+0000", 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046737510000000003 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438927+0000", 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8779000000000003e-05 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.894345020999999, 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059654479999999999, 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:35.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432945+0000", 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432987+0000", 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2401999999999998e-05 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433023+0000", 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5562999999999998e-05 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433033+0000", 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0539e-05 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433334+0000", 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000300952 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433343+0000", 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.302e-06 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438871+0000", 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055279500000000002 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438909+0000", 2024-09-17T16:02:35.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7951999999999998e-05 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=94, reqid=osd.2.0:94, at_version=79'78, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.893192704000001, 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079659940000000005, 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.385 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434097+0000", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434118+0000", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0834000000000001e-05 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440520+0000", 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064023750000000001 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440528+0000", 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6120000000000001e-06 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440597+0000", 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9429000000000002e-05 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440608+0000", 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1199000000000001e-05 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442024+0000", 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001415166 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442062+0000", 2024-09-17T16:02:35.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8469e-05 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=98, reqid=client.4560.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.879433712000001, 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25686310400000001, 2024-09-17T16:02:35.392 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:35.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447856+0000", 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.71e-07 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447869+0000", 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2795e-05 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703761+0000", 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25589224300000002 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703768+0000", 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6599999999999998e-06 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703842+0000", 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4402999999999999e-05 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703857+0000", 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4737e-05 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704682+0000", 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082487599999999999 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704718+0000", 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6519000000000001e-05 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.397 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.897280194, 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066388380000000002, 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0100000000000001e-07 2024-09-17T16:02:35.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430030+0000", 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1211000000000001e-05 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430055+0000", 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5411000000000001e-05 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430066+0000", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0635000000000001e-05 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430142+0000", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5414999999999995e-05 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430154+0000", 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2877000000000001e-05 2024-09-17T16:02:35.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436610+0000", 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064551849999999996 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436647+0000", 2024-09-17T16:02:35.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7902999999999999e-05 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 81/32 ECSubWrite(tid=241, reqid=osd.12.0:241, at_version=81'163, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.430824+0000", 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.896465081000001, 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056462109999999999, 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.403 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430824+0000", 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430824+0000", 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430824+0000", 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9999999999999999e-07 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430838+0000", 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3913999999999999e-05 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430957+0000", 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119243 2024-09-17T16:02:35.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430965+0000", 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.433878+0000", 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002913302 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.433890+0000", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1253999999999999e-05 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436421+0000", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025317569999999999 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436470+0000", 2024-09-17T16:02:35.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8531e-05 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.896280014, 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056630860000000003, 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.408 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431010+0000", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3399999999999997e-07 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431035+0000", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5491999999999999e-05 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431082+0000", 2024-09-17T16:02:35.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6853e-05 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431092+0000", 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0164e-05 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431183+0000", 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1129000000000001e-05 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431195+0000", 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1191e-05 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436653+0000", 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054587730000000001 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436672+0000", 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.855e-05 2024-09-17T16:02:35.412 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.19s4 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.878695941, 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063649290000000001, 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448594+0000", 2024-09-17T16:02:35.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448861+0000", 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026757000000000001 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448891+0000", 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9943000000000001e-05 2024-09-17T16:02:35.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448897+0000", 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.823e-06 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449029+0000", 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013207599999999999 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449043+0000", 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.414e-05 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454916+0000", 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058721440000000001 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454958+0000", 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2295000000000001e-05 2024-09-17T16:02:35.417 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 81/53 ECSubWrite(tid=155, reqid=osd.14.0:155, at_version=81'47, trim_to=0'0, roll_forward_to=77'43))", 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 15.876276061, 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065741319999999999, 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451014+0000", 2024-09-17T16:02:35.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451051+0000", 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7224999999999997e-05 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451087+0000", 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5895000000000001e-05 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451097+0000", 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451168+0000", 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0992000000000001e-05 2024-09-17T16:02:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0185e-05 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457549+0000", 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063700420000000002 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457587+0000", 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8297000000000003e-05 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 93/32 ECSubWrite(tid=133, reqid=osd.2.0:133, at_version=93'117, trim_to=0'0, roll_forward_to=92'115))", 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 3.8206345160000001, 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059325740000000004, 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.423 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506655+0000", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.16e-07 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506677+0000", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2242000000000002e-05 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506717+0000", 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0519999999999998e-05 2024-09-17T16:02:35.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506726+0000", 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3329999999999997e-06 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506996+0000", 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027085500000000002 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507008+0000", 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1080999999999999e-05 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512549+0000", 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055410959999999997 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512587+0000", 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8231e-05 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 93/53 ECSubWrite(tid=89, reqid=osd.15.0:89, at_version=93'41, trim_to=0'0, roll_forward_to=84'39))", 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 3.8194147899999997, 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076683949999999997, 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:35.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.49e-07 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507894+0000", 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9981000000000001e-05 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507975+0000", 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1248999999999994e-05 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507984+0000", 2024-09-17T16:02:35.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508051+0000", 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6963000000000002e-05 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508062+0000", 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1418e-05 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515497+0000", 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074346079999999997 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515542+0000", 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5818000000000001e-05 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.432 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:35.434 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:35.432+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:35.503 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.10 to 0.785194 (c902) 2024-09-17T16:02:35.506 INFO:tasks.rados.rados.0.smithi181.stdout:331: done (0 left) 2024-09-17T16:02:35.506 INFO:tasks.rados.rados.0.smithi181.stdout:332: delete oid 5 current snap is 38 2024-09-17T16:02:35.520 INFO:tasks.rados.rados.0.smithi181.stdout:332: done (0 left) 2024-09-17T16:02:35.520 INFO:tasks.rados.rados.0.smithi181.stdout:333: snap_remove snap 27 2024-09-17T16:02:35.565 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:35.562+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:35.589 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.590 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:35.590 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:35.590 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:35.635 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:35.633+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:35.736 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:35.734+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:35.748 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.748 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:35.748 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:35.749 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:35.749 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:35.837 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:35.835+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.866 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.995578389999999, 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019947230999999999, 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867810+0000", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867830+0000", 2024-09-17T16:02:35.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9922999999999999e-05 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881759+0000", 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01392914 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881768+0000", 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7809999999999994e-06 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881836+0000", 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8491999999999994e-05 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881849+0000", 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2418e-05 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887709+0000", 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058604850000000003 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887756+0000", 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7168e-05 2024-09-17T16:02:35.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=5, reqid=client.4560.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.987533931000002, 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012978303, 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.872 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799999999999999e-07 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875860+0000", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.02e-06 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875972+0000", 2024-09-17T16:02:35.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011225100000000001 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875986+0000", 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3526000000000001e-05 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877064+0000", 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00107855 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877074+0000", 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888801+0000", 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011726927999999999 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888832+0000", 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0750999999999997e-05 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=6, reqid=client.4560.0:52, at_version=35'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.987526723999999, 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012993864000000001, 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.877 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3199999999999999e-07 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875913+0000", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1820999999999997e-05 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876038+0000", 2024-09-17T16:02:35.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012561899999999999 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876048+0000", 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.166e-06 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877256+0000", 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012082309999999999 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877264+0000", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0399999999999993e-06 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888835+0000", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011570815 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888855+0000", 2024-09-17T16:02:35.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0040000000000001e-05 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=7, reqid=client.4560.0:53, at_version=35'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.987473327000004, 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012966596, 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.882 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.86e-07 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875921+0000", 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876450+0000", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052973099999999997 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876457+0000", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1029999999999996e-06 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879593+0000", 2024-09-17T16:02:35.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003135597 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879606+0000", 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3142000000000001e-05 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888861+0000", 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092553359999999994 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888881+0000", 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9330999999999998e-05 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=12, reqid=client.4560.0:64, at_version=35'3, trim_to=0'0, roll_forward_to=35'1))", 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.981896274999997, 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069303020000000002, 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.887 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881492+0000", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1060000000000001e-06 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881730+0000", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00023725499999999999 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882445+0000", 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071539399999999995 2024-09-17T16:02:35.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882453+0000", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2830000000000001e-06 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882512+0000", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8569000000000001e-05 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882521+0000", 2024-09-17T16:02:35.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1290000000000002e-06 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888395+0000", 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058743440000000001 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888422+0000", 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6222000000000001e-05 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.891 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:61 3.5s0 3:a2653429:::smithi18136759-21:head [append 0~344064,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.980544422000001, 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075767569999999999, 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:35.892 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 61 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882845+0000", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5650000000000001e-06 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883323+0000", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047814299999999999 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883338+0000", 2024-09-17T16:02:35.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.504e-05 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883346+0000", 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0760000000000003e-06 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5992999999999997e-05 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:35.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883498+0000", 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6426000000000001e-05 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.884493+0000", 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00099483199999999996 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889296+0000", 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048030460000000001 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890335+0000", 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001038546 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890420+0000", 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.509e-05 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.897 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:62 3.5s0 3:a2653429:::smithi18136759-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.980022587000001, 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007574038, 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 62 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883366+0000", 2024-09-17T16:02:35.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883374+0000", 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1920000000000005e-06 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883384+0000", 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0409e-05 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883390+0000", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887851+0000", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044615319999999998 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887944+0000", 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2359000000000001e-05 2024-09-17T16:02:35.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888411+0000", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00046751900000000001 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889298+0000", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088712999999999997 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890854+0000", 2024-09-17T16:02:35.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001555753 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890939+0000", 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4860999999999999e-05 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:63 3.5s0 3:a2653429:::smithi18136759-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.979985509000002, 2024-09-17T16:02:35.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010045017999999999, 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 63 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883403+0000", 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883406+0000", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2320000000000001e-06 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883414+0000", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9899999999999997e-06 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883419+0000", 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5199999999999997e-06 2024-09-17T16:02:35.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888936+0000", 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055166759999999999 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889040+0000", 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010429699999999999 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891231+0000", 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021904110000000002 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891322+0000", 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.893447+0000", 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021253399999999999 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=18, reqid=client.4560.0:65, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.972712483000002, 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080117299999999999, 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890676+0000", 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3499999999999996e-07 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890683+0000", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892226+0000", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015435869999999999 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892230+0000", 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9360000000000001e-06 2024-09-17T16:02:35.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892272+0000", 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1822e-05 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892278+0000", 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898638+0000", 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063600380000000001 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898687+0000", 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8477999999999999e-05 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.913 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=4, last_complete=35'2, committed=1, applied=1))", 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 61.971172910999996, 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013072064, 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892217+0000", 2024-09-17T16:02:35.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.063e-06 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905154+0000", 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012937357999999999 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905163+0000", 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.636e-06 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905248+0000", 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5352000000000005e-05 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905287+0000", 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8356999999999999e-05 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 47/32 ECSubWrite(tid=57, reqid=client.4560.0:282, at_version=47'18, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:35.917 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 49.929312891000002, 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069921009999999997, 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934076+0000", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934104+0000", 2024-09-17T16:02:35.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8330000000000002e-05 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934155+0000", 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1490999999999999e-05 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934165+0000", 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0121e-05 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934248+0000", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083999999999995e-05 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934258+0000", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0808999999999999e-05 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941026+0000", 2024-09-17T16:02:35.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006767421 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941067+0000", 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0978e-05 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 47/32 ECSubWrite(tid=59, reqid=client.4560.0:280, at_version=47'23, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 49.926826966, 2024-09-17T16:02:35.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074766839999999999, 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:35.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936562+0000", 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.64e-07 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936585+0000", 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3082e-05 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936627+0000", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2141999999999997e-05 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936636+0000", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1509999999999993e-06 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937201+0000", 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00056518400000000004 2024-09-17T16:02:35.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937210+0000", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1050000000000001e-06 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944001+0000", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067911480000000003 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944037+0000", 2024-09-17T16:02:35.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5908000000000002e-05 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 43.513141367999999, 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007077552, 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.927 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350272+0000", 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5619e-05 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350409+0000", 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013671899999999999 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350418+0000", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350893+0000", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047499899999999998 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350904+0000", 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0977e-05 2024-09-17T16:02:35.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357278+0000", 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063733080000000003 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357324+0000", 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6180999999999997e-05 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 55/32 ECSubWrite(tid=97, reqid=client.4560.0:353, at_version=55'45, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 42.488498317000001, 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006702225, 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.374889+0000", 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375005+0000", 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011503800000000001 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375080+0000", 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5148999999999995e-05 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375088+0000", 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6929999999999996e-06 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375199+0000", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011076100000000001 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.375211+0000", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1885000000000001e-05 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381558+0000", 2024-09-17T16:02:35.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063466750000000004 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381591+0000", 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3747000000000001e-05 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 64/53 ECSubWriteReply(tid=28, last_complete=64'12, committed=1, applied=1))", 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:35.936 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.502430828999998, 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79147448499999995, 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.937 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.75e-07 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360961+0000", 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9550000000000002e-06 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152290+0000", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79132856699999998 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152299+0000", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152377+0000", 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8317000000000002e-05 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152431+0000", 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3773000000000002e-05 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 70/32 ECSubWrite(tid=171, reqid=client.4560.0:509, at_version=70'105, trim_to=0'0, roll_forward_to=69'101))", 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 27.485829706000001, 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14998041000000001, 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.940 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.941 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377559+0000", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7800000000000001e-07 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377579+0000", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0820999999999999e-05 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526840+0000", 2024-09-17T16:02:35.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14926072600000001 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526849+0000", 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2499999999999995e-06 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526946+0000", 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7052000000000005e-05 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.943 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526954+0000", 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1940000000000001e-06 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.527499+0000", 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054546100000000001 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.527538+0000", 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9227999999999998e-05 2024-09-17T16:02:35.944 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 81/53 ECSubWrite(tid=76, reqid=osd.15.0:74, at_version=0'0, trim_to=0'0, roll_forward_to=81'33))", 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.416470326999999, 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082149000000000007, 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446918+0000", 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446930+0000", 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1639e-05 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446971+0000", 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1591000000000002e-05 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.947 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446979+0000", 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0360000000000006e-06 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.447063+0000", 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3627000000000004e-05 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.447073+0000", 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.948 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455094+0000", 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080202699999999995 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455132+0000", 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8513000000000002e-05 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.949 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 86/65 ECSubWrite(tid=287, reqid=client.4560.0:768, at_version=86'56, trim_to=0'0, roll_forward_to=86'53))", 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.370258314999999, 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79617869100000005, 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493130+0000", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493150+0000", 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0514999999999999e-05 2024-09-17T16:02:35.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288281+0000", 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.795130106 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288297+0000", 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6243e-05 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288454+0000", 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015702500000000001 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288488+0000", 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4517000000000002e-05 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289268+0000", 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000779829 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289308+0000", 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9774999999999998e-05 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 93/32 ECSubWrite(tid=292, reqid=osd.12.0:291, at_version=0'0, trim_to=0'0, roll_forward_to=93'205))", 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.3581146979999996, 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.008291484, 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.954 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505274+0000", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505285+0000", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1688000000000001e-05 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505321+0000", 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5126000000000002e-05 2024-09-17T16:02:35.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505329+0000", 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1340000000000004e-06 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505453+0000", 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012396 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505464+0000", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1223000000000001e-05 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513528+0000", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080639379999999997 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513564+0000", 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6486000000000003e-05 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.958 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 93/32 ECSubWrite(tid=293, reqid=osd.12.0:293, at_version=93'207, trim_to=0'0, roll_forward_to=93'205))", 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4.3572014069999998, 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074650159999999997, 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:35.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506187+0000", 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.23e-07 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506215+0000", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7881e-05 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506258+0000", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3362999999999998e-05 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506266+0000", 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3829999999999992e-06 2024-09-17T16:02:35.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507457+0000", 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001191012 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507468+0000", 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0492999999999999e-05 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:35.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513624+0000", 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061560909999999998 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513651+0000", 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.707e-05 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:35.963 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:35.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:35.965 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:35.962+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:36.065 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:36.064+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:36.098 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:36.098 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:36.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:36.116 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:36.116 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:36.116 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:36.116 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:36.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:36.166 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:36.164+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:36.268 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:36.266+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:36.273 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:36.273 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:36.273 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:36.273 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:36.273 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:36.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:36.368 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:36.365+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:24 3.ds0 3:be8168ea:::smithi18136759-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.532487832999998, 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.04871255, 2024-09-17T16:02:36.392 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 24 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:36.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6700000000000003e-07 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856438+0000", 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0089999999999999e-06 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856446+0000", 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1240000000000005e-06 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856452+0000", 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8490000000000002e-06 2024-09-17T16:02:36.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858431+0000", 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019785380000000002 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858505+0000", 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4269000000000003e-05 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876493+0000", 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017988067999999999 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876576+0000", 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2981999999999999e-05 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905147+0000", 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028571144 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:28 3.ds0 3:b6a72c22:::smithi18136759-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:36.398 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.532461013000002, 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043697429000000003, 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 28 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856462+0000", 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856586+0000", 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012445300000000001 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5877000000000001e-05 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856609+0000", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5869999999999998e-06 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858561+0000", 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019522739999999999 2024-09-17T16:02:36.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858680+0000", 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118678 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859415+0000", 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073567700000000001 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864061+0000", 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046461560000000002 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900068+0000", 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036006330000000003 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900159+0000", 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0970999999999993e-05 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:29 3.ds0 3:b6a72c22:::smithi18136759-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.532302389999998, 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043754779000000001, 2024-09-17T16:02:36.405 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 29 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.406 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856621+0000", 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2099999999999996e-07 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856624+0000", 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7000000000000002e-06 2024-09-17T16:02:36.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856631+0000", 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856638+0000", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859904+0000", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032664679999999998 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859980+0000", 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5711999999999996e-05 2024-09-17T16:02:36.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860364+0000", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038396600000000002 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864272+0000", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003907888 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900273+0000", 2024-09-17T16:02:36.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036001247 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900375+0000", 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101535 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:30 3.ds0 3:b6a72c22:::smithi18136759-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:36.412 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.532276564, 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.048780709999999998, 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 30 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:36.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4499999999999997e-07 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856657+0000", 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856664+0000", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.148e-06 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856673+0000", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.098e-06 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860632+0000", 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039593440000000001 2024-09-17T16:02:36.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860703+0000", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0197000000000006e-05 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900473+0000", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039770158 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900637+0000", 2024-09-17T16:02:36.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000164356 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905426+0000", 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047893579999999996 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:40 3.ds0 3:b22f7774:::smithi18136759-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:36.420 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.527273211000001, 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039805350000000003, 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 40 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:36.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861651+0000", 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.525e-06 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861832+0000", 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018106599999999999 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861846+0000", 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4497e-05 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861854+0000", 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0390000000000005e-06 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.791e-05 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862054+0000", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014139799999999999 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862812+0000", 2024-09-17T16:02:36.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000758028 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867004+0000", 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004192594 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901412+0000", 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034408162999999999 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901454+0000", 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2129999999999998e-05 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:41 3.ds0 3:b22f7774:::smithi18136759-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.527046904999999, 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039711297, 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.426 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 41 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861876+0000", 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4e-07 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861880+0000", 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861887+0000", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0890000000000002e-06 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861893+0000", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8649999999999998e-06 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863320+0000", 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001427212 2024-09-17T16:02:36.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863412+0000", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1540999999999997e-05 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863845+0000", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043304900000000002 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867006+0000", 2024-09-17T16:02:36.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031615369999999999 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901546+0000", 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034539746000000003 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901587+0000", 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0802e-05 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:42 3.ds0 3:b22f7774:::smithi18136759-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.527014411000003, 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043739746000000003, 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 42 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.432 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861909+0000", 2024-09-17T16:02:36.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9400000000000005e-07 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3019999999999999e-06 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861919+0000", 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861925+0000", 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8529999999999997e-06 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864137+0000", 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022118649999999999 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864205+0000", 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8396e-05 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901627+0000", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037421487000000003 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901674+0000", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7577000000000002e-05 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905648+0000", 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039733169999999996 2024-09-17T16:02:36.436 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:52 3.ds0 3:bf204631:::smithi18136759-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.518460722, 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.031550519999999999, 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.437 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 52 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870462+0000", 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870463+0000", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.375e-06 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870715+0000", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025214399999999999 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870733+0000", 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7407000000000001e-05 2024-09-17T16:02:36.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870749+0000", 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6682000000000001e-05 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870815+0000", 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5939999999999995e-05 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870975+0000", 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015933899999999999 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.871691+0000", 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071632 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.874021+0000", 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023304419999999998 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901948+0000", 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.027927041999999999 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902012+0000", 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3829000000000002e-05 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:54 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:36.442 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.518119603000002, 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035090831000000003, 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 54 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:36.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9800000000000004e-07 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870806+0000", 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3549999999999999e-06 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870813+0000", 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1539999999999996e-06 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870819+0000", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0700000000000003e-06 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873071+0000", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022519200000000001 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873154+0000", 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3746999999999996e-05 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902680+0000", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029525513 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902767+0000", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6837999999999998e-05 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905894+0000", 2024-09-17T16:02:36.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031267360000000002 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 46/32 ECSubWriteReply(tid=54, last_complete=46'16, committed=1, applied=1))", 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 51.450073388, 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62914934300000003, 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.448 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938850+0000", 2024-09-17T16:02:36.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.06e-07 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938855+0000", 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0010000000000001e-06 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567797+0000", 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62894212500000002 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567814+0000", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6847e-05 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567931+0000", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011766500000000001 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567998+0000", 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6999e-05 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.451 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:284 3.as0 3:5583c93f:::smithi18136759-40:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.45540441, 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.386964417, 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.452 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 284 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933519+0000", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933522+0000", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4010000000000001e-06 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933536+0000", 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4269e-05 2024-09-17T16:02:36.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933544+0000", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.576e-06 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938388+0000", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048441300000000003 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938478+0000", 2024-09-17T16:02:36.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9975000000000005e-05 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939194+0000", 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071552499999999995 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944047+0000", 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048533990000000004 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320427+0000", 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37638003799999997 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320482+0000", 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5390000000000003e-05 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:358 3.ds0 3:b27fc74f:::smithi18136759-24:head [copy-from ver 28] snapc 11=[11,10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:36.457 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 43.012805716000003, 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.040742296999999997, 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 358 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376121+0000", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1679999999999997e-06 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376135+0000", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3054000000000001e-05 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376141+0000", 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.443e-06 2024-09-17T16:02:36.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381260+0000", 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005118671 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381327+0000", 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7288999999999999e-05 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.392972+0000", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011645071999999999 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.396350+0000", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033780709999999999 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416805+0000", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020454785999999999 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416859+0000", 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4045999999999997e-05 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 63/32 ECSubWriteReply(tid=124, last_complete=63'41, committed=1, applied=1))", 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.961593628000003, 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45322852000000002, 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.463 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:36.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427336+0000", 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4069999999999998e-06 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880036+0000", 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45270011199999999 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880042+0000", 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7950000000000004e-06 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880109+0000", 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7659999999999999e-05 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880557+0000", 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044786499999999998 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.466 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:509 3.ds0 3:bf204631:::smithi18136759-18:head [setxattr _header (57) in=7b] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.014081152999999, 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91880832899999998, 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.467 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 509 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374842+0000", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9999999999999995e-07 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374847+0000", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.724e-06 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374855+0000", 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4400000000000005e-06 2024-09-17T16:02:36.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374862+0000", 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377155+0000", 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022931840000000002 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377217+0000", 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2512000000000001e-05 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377509+0000", 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000291057 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.381196+0000", 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036871849999999999 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293578+0000", 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91238186700000001 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293650+0000", 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1950000000000001e-05 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:510 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.014051267999999, 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.92014742599999999, 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 510 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.473 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374872+0000", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374875+0000", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.083e-06 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374883+0000", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374890+0000", 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9469999999999997e-06 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378503+0000", 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036131100000000001 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378624+0000", 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121218 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294084+0000", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91545977899999997 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294190+0000", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105797 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.295019+0000", 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082872900000000005 2024-09-17T16:02:36.477 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 70/32 ECSubWriteReply(tid=171, last_complete=70'105, committed=1, applied=1))", 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.010348338, 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91541347900000003, 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.478 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378575+0000", 2024-09-17T16:02:36.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378579+0000", 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0289999999999999e-06 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293447+0000", 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91486866300000003 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293456+0000", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5040000000000002e-06 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293520+0000", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3893999999999995e-05 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293987+0000", 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000467725 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.481 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/32 ECSubWriteReply(tid=246, last_complete=81'169, committed=1, applied=1))", 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 16.933786193, 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.121075089, 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455137+0000", 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0999999999999997e-07 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455142+0000", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9250000000000003e-06 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575575+0000", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.120433441 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575581+0000", 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5720000000000002e-06 2024-09-17T16:02:36.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575664+0000", 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3115000000000006e-05 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.576211+0000", 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054722600000000005 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 85/32 ECSubWriteReply(tid=275, last_complete=85'195, committed=1, applied=1))", 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.920252594000001, 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87648519400000002, 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:36.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3600000000000003e-07 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468673+0000", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0400000000000001e-06 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344179+0000", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87550525700000004 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344191+0000", 2024-09-17T16:02:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2237e-05 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344247+0000", 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6391000000000003e-05 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.345155+0000", 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090763300000000003 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:768 3.as0 3:5ccfabf6:::smithi18136759-44:head [setxattr _header (57) in=7b] snapc 20=[20,1f,1e,1d,1c,19,18,16,10,c,6] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.900568706, 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.801372309, 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 768 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.490 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5199999999999996e-07 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488358+0000", 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4479999999999999e-06 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488366+0000", 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8380000000000002e-06 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488370+0000", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2420000000000002e-06 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492136+0000", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037664959999999998 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492251+0000", 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114357 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492748+0000", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00049682299999999997 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.494817+0000", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002069286 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289552+0000", 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79473559500000002 2024-09-17T16:02:36.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289726+0000", 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017347200000000001 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:769 3.as0 3:5ccfabf6:::smithi18136759-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 11.900546216, 2024-09-17T16:02:36.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80260884799999999, 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 769 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:36.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488377+0000", 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7099999999999997e-07 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:36.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488380+0000", 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7069999999999998e-06 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488386+0000", 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4970000000000001e-06 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488389+0000", 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9869999999999999e-06 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493011+0000", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046225939999999998 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493109+0000", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8202999999999995e-05 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290108+0000", 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79699814300000005 2024-09-17T16:02:36.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290237+0000", 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012928200000000001 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290985+0000", 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000748064 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:36.500 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:36.501 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:36.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:36.503 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:36.500+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:36.507 INFO:tasks.rados.rados.0.smithi181.stdout:333: done (0 left) 2024-09-17T16:02:36.507 INFO:tasks.rados.rados.0.smithi181.stdout:334: snap_remove snap 30 2024-09-17T16:02:36.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:36.602 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:36.600+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:36.632 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:36.633 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:36.633 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:36.633 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:36.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:36.704 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:36.702+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:36.748 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:36.749 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:36.749 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:36.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:36.805 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:36.802+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:238 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.940829585000003, 2024-09-17T16:02:36.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011165420000000001, 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 238 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921315+0000", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921316+0000", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3480000000000001e-06 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921322+0000", 2024-09-17T16:02:36.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7729999999999996e-06 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921333+0000", 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0779e-05 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921339+0000", 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9699999999999996e-06 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921387+0000", 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8588000000000003e-05 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921462+0000", 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4530999999999995e-05 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922070+0000", 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060823699999999997 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.869 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923628+0000", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001558219 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932393+0000", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087645399999999995 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932480+0000", 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7435000000000006e-05 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.870 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:239 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 5] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.940790004999997, 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011291409000000001, 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.871 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 239 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921354+0000", 2024-09-17T16:02:36.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921355+0000", 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4000000000000002e-07 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921358+0000", 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.568e-06 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921365+0000", 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8040000000000004e-06 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921371+0000", 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.536e-06 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922590+0000", 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012196679999999999 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.922675+0000", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4715000000000005e-05 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923334+0000", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065929300000000003 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925081+0000", 2024-09-17T16:02:36.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001747024 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932583+0000", 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0075011599999999998 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932646+0000", 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3200999999999993e-05 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:240 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.940751794999997, 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011490029000000001, 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 240 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.877 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921393+0000", 2024-09-17T16:02:36.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7899999999999999e-07 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921396+0000", 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.52e-06 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921402+0000", 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6309999999999997e-06 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.921408+0000", 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5690000000000004e-06 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923661+0000", 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022530979999999998 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.923874+0000", 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021281499999999999 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.924326+0000", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045228500000000002 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.925665+0000", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013387049999999999 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932831+0000", 2024-09-17T16:02:36.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071659510000000003 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.932883+0000", 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1975999999999998e-05 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:272 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:36.882 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.936463895999999, 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046207150000000001, 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 272 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.883 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925680+0000", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925682+0000", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7060000000000001e-06 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925690+0000", 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3950000000000003e-06 2024-09-17T16:02:36.884 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925706+0000", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.611e-05 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925713+0000", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9809999999999997e-06 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2625000000000003e-05 2024-09-17T16:02:36.885 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925900+0000", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011498 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.926711+0000", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00081025800000000001 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929013+0000", 2024-09-17T16:02:36.886 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023021669999999999 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930254+0000", 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012415900000000001 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.930301+0000", 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6903000000000001e-05 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.887 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:273 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 8] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.936414978000002, 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070115810000000002, 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 273 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.888 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925729+0000", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925730+0000", 2024-09-17T16:02:36.889 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8900000000000001e-07 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925733+0000", 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3900000000000002e-06 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925740+0000", 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6610000000000003e-06 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.890 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925752+0000", 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2038000000000001e-05 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927223+0000", 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014708670000000001 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927310+0000", 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7535999999999997e-05 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.891 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.927853+0000", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054293600000000005 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929016+0000", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001162445 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932681+0000", 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003665092 2024-09-17T16:02:36.892 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.932741+0000", 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0327e-05 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:274 3.4s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.936364595999997, 2024-09-17T16:02:36.893 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011049606999999999, 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 274 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:36.894 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925780+0000", 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.84e-07 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.895 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925785+0000", 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.425e-06 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925790+0000", 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2100000000000001e-06 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.925793+0000", 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3409999999999999e-06 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.896 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928169+0000", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023761490000000001 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928252+0000", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2699000000000002e-05 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.928677+0000", 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042531099999999999 2024-09-17T16:02:36.897 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.929281+0000", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060397100000000002 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936793+0000", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00751164 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.936829+0000", 2024-09-17T16:02:36.898 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6477000000000003e-05 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.fs6 54/32 ECSubWrite(tid=53, reqid=client.4560.0:344, at_version=54'24, trim_to=0'0, roll_forward_to=52'22))", 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.519895525999999, 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036631329999999998, 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.899 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.900 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342249+0000", 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8099999999999998e-07 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342277+0000", 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7398000000000001e-05 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342817+0000", 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054081699999999997 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.901 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342835+0000", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7779999999999999e-05 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342920+0000", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4445999999999998e-05 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.342932+0000", 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2283000000000001e-05 2024-09-17T16:02:36.902 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345862+0000", 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002929577 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:52.345912+0000", 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0451000000000002e-05 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.903 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 64/53 ECSubWrite(tid=41, reqid=osd.15.0:41, at_version=64'23, trim_to=0'0, roll_forward_to=62'21))", 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.524051542999999, 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003803233, 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.904 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338093+0000", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338094+0000", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1699999999999997e-07 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338127+0000", 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3801999999999999e-05 2024-09-17T16:02:36.905 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338724+0000", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059661000000000002 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338737+0000", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3168e-05 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338818+0000", 2024-09-17T16:02:36.906 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0558999999999999e-05 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.338830+0000", 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2686e-05 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341825+0000", 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029948940000000001 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.907 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.341896+0000", 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0797000000000007e-05 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.516572707000002, 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047088649999999996, 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.908 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.909 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345572+0000", 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3500000000000002e-07 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345597+0000", 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4491e-05 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346172+0000", 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00057511800000000001 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346180+0000", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3480000000000004e-06 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346420+0000", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023985200000000001 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346431+0000", 2024-09-17T16:02:36.911 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1372000000000001e-05 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350228+0000", 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037965479999999998 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350281+0000", 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2800999999999998e-05 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.912 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s5 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.516552163999997, 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040542870000000002, 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.913 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4600000000000001e-07 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.914 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345620+0000", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7722999999999999e-05 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345697+0000", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7019000000000002e-05 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345707+0000", 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:36.915 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345799+0000", 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2009999999999995e-05 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345809+0000", 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0036e-05 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.916 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349602+0000", 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037928200000000001 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349647+0000", 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4552000000000003e-05 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s5 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:36.917 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.516521150999999, 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047402829999999996, 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.918 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345623+0000", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345624+0000", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345653+0000", 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9376000000000001e-05 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.919 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345739+0000", 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5702e-05 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345745+0000", 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2929999999999997e-06 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.920 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346167+0000", 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042231199999999999 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346178+0000", 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0627e-05 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350309+0000", 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0041311509999999996 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.921 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350364+0000", 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4466000000000003e-05 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.13s2 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.515870190999998, 2024-09-17T16:02:36.922 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039770489999999999, 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.923 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346274+0000", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1400000000000001e-07 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346300+0000", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5800999999999999e-05 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346608+0000", 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030764399999999998 2024-09-17T16:02:36.924 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346618+0000", 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.747e-06 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346718+0000", 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010035900000000001 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.925 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3066e-05 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350161+0000", 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034297310000000001 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350251+0000", 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0487000000000003e-05 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.926 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.17s3 73/65 ECSubWrite(tid=172, reqid=client.4560.0:581, at_version=73'38, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.448605385, 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.93455132600000002, 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:36.927 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413539+0000", 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413540+0000", 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2600000000000002e-07 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.928 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:11.413561+0000", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1708999999999998e-05 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347419+0000", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.933857559 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347428+0000", 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9579999999999996e-06 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.929 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347484+0000", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6483000000000001e-05 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.347492+0000", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9100000000000005e-06 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.348050+0000", 2024-09-17T16:02:36.930 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055774199999999996 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.348090+0000", 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0238999999999999e-05 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:631 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.433469539000001, 2024-09-17T16:02:36.931 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085792290000000007, 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 631 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:36.932 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428675+0000", 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428677+0000", 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4420000000000001e-06 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.933 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428687+0000", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.577e-06 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428709+0000", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1781999999999999e-05 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428720+0000", 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.094e-05 2024-09-17T16:02:36.934 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428800+0000", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0217999999999997e-05 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428809+0000", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431022+0000", 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022127649999999998 2024-09-17T16:02:36.935 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431120+0000", 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7845999999999998e-05 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432156+0000", 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001036896 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.936 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435767+0000", 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003610753 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.437165+0000", 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013976769999999999 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.437254+0000", 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9155999999999998e-05 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.937 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:632 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [rollback 6] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.433398092000001, 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.009419696, 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.938 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 632 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428746+0000", 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.939 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428747+0000", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.68e-07 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428750+0000", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1810000000000001e-06 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428760+0000", 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0198e-05 2024-09-17T16:02:36.940 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428766+0000", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428881+0000", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011492 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428894+0000", 2024-09-17T16:02:36.941 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3129000000000001e-05 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432631+0000", 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003736971 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432729+0000", 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7295999999999998e-05 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.942 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433311+0000", 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058199599999999999 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435769+0000", 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024582969999999999 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438084+0000", 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023153480000000001 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.943 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438166+0000", 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1771e-05 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:633 3.14s0 3:2dc8b6cd:::smithi18136759-26:head [append 0~0] snapc 1b=[19,18,16,10,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.433349639999999, 2024-09-17T16:02:36.944 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010126513, 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 633 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.945 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428795+0000", 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428796+0000", 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0599999999999999e-07 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.946 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428800+0000", 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6330000000000004e-06 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428813+0000", 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2906e-05 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428821+0000", 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5619999999999998e-06 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.947 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428921+0000", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010017 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.428927+0000", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9889999999999997e-06 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433618+0000", 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046906439999999999 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.948 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433715+0000", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7182000000000006e-05 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434216+0000", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050111399999999999 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436519+0000", 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023034869999999999 2024-09-17T16:02:36.949 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438845+0000", 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023260189999999999 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.438921+0000", 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5901000000000004e-05 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.950 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs5 81/32 ECSubWrite(tid=102, reqid=osd.0.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=81'49))", 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.415286622, 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036110410000000002, 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.951 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446858+0000", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446859+0000", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4799999999999997e-07 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446872+0000", 2024-09-17T16:02:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3324e-05 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446904+0000", 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2388000000000002e-05 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446913+0000", 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4910000000000005e-06 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447549+0000", 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00063597399999999998 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447560+0000", 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1453e-05 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.450429+0000", 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028682650000000001 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.450469+0000", 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0297999999999999e-05 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s5 93/53 ECSubWrite(tid=62, reqid=osd.8.0:62, at_version=93'32, trim_to=0'0, roll_forward_to=84'30))", 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:31.508596+0000", 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 5.35354888, 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004560021, 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508596+0000", 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508596+0000", 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508596+0000", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2000000000000001e-07 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508616+0000", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9723000000000001e-05 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508654+0000", 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8359000000000003e-05 2024-09-17T16:02:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508662+0000", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4679999999999992e-06 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508718+0000", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5387999999999999e-05 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.508731+0000", 2024-09-17T16:02:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3219e-05 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.513116+0000", 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043845350000000002 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.513156+0000", 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0009000000000003e-05 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.3s2 98/53 ECSubWrite(tid=91, reqid=osd.15.0:91, at_version=98'43, trim_to=0'0, roll_forward_to=93'41))", 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.511205+0000", 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.35093913700000001, 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058879759999999996, 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511205+0000", 2024-09-17T16:02:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511205+0000", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511205+0000", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0200000000000001e-07 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511233+0000", 2024-09-17T16:02:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7365999999999999e-05 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513012+0000", 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001779208 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513022+0000", 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0081000000000001e-05 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513382+0000", 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035991899999999999 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513393+0000", 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1399e-05 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.517048+0000", 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036543750000000001 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.517093+0000", 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5426000000000002e-05 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s5 98/53 ECSubWrite(tid=69, reqid=osd.8.0:69, at_version=98'32, trim_to=0'0, roll_forward_to=97'30))", 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.511610+0000", 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.35053427399999998, 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043608120000000004, 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511610+0000", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511610+0000", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511611+0000", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7700000000000003e-07 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511650+0000", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9051000000000003e-05 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511689+0000", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8899000000000001e-05 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511698+0000", 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2490000000000007e-06 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512733+0000", 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001035221 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512755+0000", 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1372000000000001e-05 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515927+0000", 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031723810000000002 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515971+0000", 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3761999999999999e-05 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:36.968 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:36.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:36.970 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:36.968+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:37.070 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:37.069+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:37.078 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:37.079 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:37.172 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:37.169+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:37.234 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.234 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:37.234 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:37.234 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:37.234 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:37.272 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:37.270+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:37.350 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.350 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s4 44/32 ECSubWrite(tid=34, reqid=client.4560.0:241, at_version=44'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.419266198000003, 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003220185, 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928634+0000", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928636+0000", 2024-09-17T16:02:37.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2279999999999999e-06 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.928980+0000", 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034461600000000001 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929020+0000", 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9878e-05 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929029+0000", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1670000000000005e-06 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929097+0000", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7960000000000007e-05 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.929107+0000", 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0063e-05 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931799+0000", 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026920310000000001 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.931855+0000", 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 46/32 ECSubWrite(tid=52, reqid=osd.12.0:52, at_version=46'19, trim_to=0'0, roll_forward_to=41'17))", 2024-09-17T16:02:37.355 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 52.422514382000003, 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95295290099999996, 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925386+0000", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925387+0000", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1100000000000001e-07 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:44.925424+0000", 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7215999999999997e-05 2024-09-17T16:02:37.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877344+0000", 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95191999800000004 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877367+0000", 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3655999999999998e-05 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877525+0000", 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015736299999999999 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.877534+0000", 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9339999999999995e-06 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878304+0000", 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00077063299999999995 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.878339+0000", 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4690000000000002e-05 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 47/32 ECSubWrite(tid=56, reqid=client.4560.0:278, at_version=47'21, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 51.415688766000002, 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032216739999999999, 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.360 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932212+0000", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.77e-07 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932813+0000", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060064099999999996 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932864+0000", 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0909000000000003e-05 2024-09-17T16:02:37.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932877+0000", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3390000000000001e-05 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932963+0000", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6266000000000004e-05 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.932975+0000", 2024-09-17T16:02:37.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.185e-05 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935383+0000", 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00240742 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:45.935433+0000", 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0821e-05 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.7s5 52/32 ECSubWrite(tid=78, reqid=osd.13.0:78, at_version=52'18, trim_to=0'0, roll_forward_to=50'16))", 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 46.996364358000001, 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017347922000000002, 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351536+0000", 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351537+0000", 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7800000000000002e-07 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.351558+0000", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1399e-05 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367847+0000", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016289428000000002 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367858+0000", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0835e-05 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367938+0000", 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0104000000000004e-05 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.367950+0000", 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1153e-05 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368846+0000", 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089654899999999996 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:50.368884+0000", 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8075999999999999e-05 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 43.954770250000003, 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047653560000000001, 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393130+0000", 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.393131+0000", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9600000000000008e-07 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394022+0000", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00089089400000000004 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394066+0000", 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4193000000000002e-05 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394089+0000", 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2608000000000001e-05 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394165+0000", 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5638999999999999e-05 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.394175+0000", 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0277e-05 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397846+0000", 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036707889999999998 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397896+0000", 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9960000000000003e-05 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.12s2 64/61 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=64'34, trim_to=0'0, roll_forward_to=62'32))", 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.010619824999999, 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035966639999999999, 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:37.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337281+0000", 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.04e-07 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.375 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337334+0000", 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3352e-05 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337390+0000", 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5319999999999999e-05 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337400+0000", 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0655999999999999e-05 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.376 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337509+0000", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000108571 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.337523+0000", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4211e-05 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340822+0000", 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032988689999999998 2024-09-17T16:02:37.377 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.340877+0000", 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5481e-05 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.378 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s2 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.002588099, 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035907539999999998, 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345312+0000", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345313+0000", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345362+0000", 2024-09-17T16:02:37.380 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8804000000000002e-05 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345410+0000", 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8133999999999997e-05 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345419+0000", 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6039999999999999e-06 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.381 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345499+0000", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0173000000000006e-05 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345510+0000", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0913999999999999e-05 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348833+0000", 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033223749999999998 2024-09-17T16:02:37.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348903+0000", 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0499000000000004e-05 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.001190813999997, 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034797959999999998, 2024-09-17T16:02:37.383 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346710+0000", 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346711+0000", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6600000000000005e-07 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346735+0000", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4652e-05 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346773+0000", 2024-09-17T16:02:37.385 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7855999999999998e-05 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346782+0000", 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5599999999999994e-06 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346854+0000", 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2512e-05 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.386 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346865+0000", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0957e-05 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350146+0000", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032811709999999998 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350190+0000", 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3222000000000001e-05 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.387 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 76/32 ECSubWrite(tid=207, reqid=client.4560.0:597, at_version=76'132, trim_to=0'0, roll_forward_to=75'130))", 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 22.938755052000001, 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045991310000000002, 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.388 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.409146+0000", 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:37.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410426+0000", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012808170000000001 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410463+0000", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6449000000000003e-05 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410471+0000", 2024-09-17T16:02:37.390 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8420000000000005e-06 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410556+0000", 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5426999999999994e-05 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.410574+0000", 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7306999999999999e-05 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.391 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413700+0000", 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031262460000000001 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:14.413745+0000", 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4910000000000002e-05 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:37.392 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.915932353999999, 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036738819999999998, 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.393 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431968+0000", 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431969+0000", 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3500000000000002e-07 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.431996+0000", 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7461e-05 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.394 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432037+0000", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0552000000000001e-05 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432059+0000", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2345e-05 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432134+0000", 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5159000000000003e-05 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.432145+0000", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0875e-05 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435607+0000", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034618510000000002 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.435642+0000", 2024-09-17T16:02:37.396 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5203999999999997e-05 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=225, reqid=client.4560.0:627, at_version=79'152, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.914886112000001, 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003614744, 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433014+0000", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433015+0000", 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:37.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433165+0000", 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149612 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433313+0000", 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014852600000000001 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.399 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433322+0000", 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4770000000000003e-06 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433429+0000", 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010759099999999999 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433441+0000", 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1323e-05 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436598+0000", 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031569749999999998 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436629+0000", 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1473000000000002e-05 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.401 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 19.914729125000001, 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034963870000000001, 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.402 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433171+0000", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433181+0000", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7489999999999993e-06 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433276+0000", 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4340999999999997e-05 2024-09-17T16:02:37.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433357+0000", 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1524000000000001e-05 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.433364+0000", 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5710000000000003e-06 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.404 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434090+0000", 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072590199999999995 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.434157+0000", 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7516999999999997e-05 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436645+0000", 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024882789999999999 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:17.436668+0000", 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2504e-05 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=240, reqid=osd.12.0:240, at_version=81'161, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.918371438000001, 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003961633, 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.406 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.407 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429529+0000", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429579+0000", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9345999999999998e-05 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430294+0000", 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071481400000000003 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.408 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430305+0000", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1618e-05 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430487+0000", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018190799999999999 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430503+0000", 2024-09-17T16:02:37.409 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.643e-05 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433445+0000", 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029419300000000002 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433491+0000", 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5348e-05 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.410 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 81/32 ECSubWrite(tid=241, reqid=osd.12.0:241, at_version=81'163, trim_to=0'0, roll_forward_to=80'159))", 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.918318211999999, 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0053980119999999998, 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:37.411 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429582+0000", 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429583+0000", 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2399999999999999e-07 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.429601+0000", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8195000000000001e-05 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430343+0000", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00074222800000000005 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.430351+0000", 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5329999999999997e-06 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.413 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431146+0000", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079573700000000003 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431157+0000", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0696999999999999e-05 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434934+0000", 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037763610000000002 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.414 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434980+0000", 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6836999999999999e-05 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.916595434000001, 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036787539999999998, 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.415 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.416 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431305+0000", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.29e-07 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431329+0000", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3555000000000001e-05 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431411+0000", 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1885999999999995e-05 2024-09-17T16:02:37.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431421+0000", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0139999999999999e-05 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431508+0000", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7016999999999997e-05 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431520+0000", 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2160999999999999e-05 2024-09-17T16:02:37.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434938+0000", 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034178939999999999 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.434984+0000", 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5871999999999997e-05 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.419 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 17.916568964, 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040176439999999999, 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.420 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431332+0000", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431352+0000", 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0449e-05 2024-09-17T16:02:37.421 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431581+0000", 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022874899999999999 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.431589+0000", 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0299999999999994e-06 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.422 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.432989+0000", 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014002800000000001 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.433001+0000", 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2004999999999999e-05 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435321+0000", 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002319883 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.423 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.435349+0000", 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8008e-05 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.ds6 98/32 ECSubWrite(tid=299, reqid=osd.12.0:299, at_version=98'215, trim_to=0'0, roll_forward_to=93'211))", 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.507720+0000", 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.840180543, 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003377021, 2024-09-17T16:02:37.424 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507720+0000", 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507720+0000", 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507721+0000", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0529999999999999e-06 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507751+0000", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0337000000000001e-05 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507790+0000", 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9022e-05 2024-09-17T16:02:37.426 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507800+0000", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9320000000000008e-06 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507902+0000", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101635 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.507914+0000", 2024-09-17T16:02:37.427 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1946999999999999e-05 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511060+0000", 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031457070000000002 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511097+0000", 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7388000000000002e-05 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.428 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s2 98/53 ECSubWrite(tid=68, reqid=osd.8.0:68, at_version=98'34, trim_to=0'0, roll_forward_to=93'32))", 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.511472+0000", 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.83642898200000004, 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043037869999999999, 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511472+0000", 2024-09-17T16:02:37.429 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511472+0000", 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511472+0000", 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5100000000000001e-07 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.430 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511500+0000", 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7773e-05 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511593+0000", 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2932999999999999e-05 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511601+0000", 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9589999999999995e-06 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511668+0000", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7760000000000002e-05 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511680+0000", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1606999999999999e-05 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515732+0000", 2024-09-17T16:02:37.432 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040524530000000001 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515775+0000", 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3050999999999998e-05 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.433 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s2 98/53 ECSubWrite(tid=69, reqid=osd.8.0:69, at_version=98'32, trim_to=0'0, roll_forward_to=97'30))", 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.512112+0000", 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.83578896400000002, 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039985280000000003, 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512112+0000", 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.434 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512112+0000", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512113+0000", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2500000000000004e-07 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512140+0000", 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7616000000000001e-05 2024-09-17T16:02:37.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512183+0000", 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2857e-05 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512192+0000", 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2750000000000001e-06 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.436 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512311+0000", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118452 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512324+0000", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2787e-05 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.516035+0000", 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037110530000000002 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.516110+0000", 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5562999999999995e-05 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.es6 98/32 ECSubWrite(tid=138, reqid=osd.2.0:138, at_version=98'123, trim_to=0'0, roll_forward_to=97'121))", 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.512203+0000", 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 0.83569759899999996, 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039091200000000003, 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512203+0000", 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512203+0000", 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.439 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512203+0000", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8200000000000001e-07 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512229+0000", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5802000000000001e-05 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512366+0000", 2024-09-17T16:02:37.440 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136723 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512373+0000", 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1319999999999996e-06 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512434+0000", 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0622999999999999e-05 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.441 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512445+0000", 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.155e-05 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.516063+0000", 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0036180069999999999 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.516112+0000", 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9001000000000001e-05 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.442 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.443 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.443 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:37.446 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:37.441+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:37.506 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:37.510 INFO:tasks.rados.rados.0.smithi181.stdout:334: done (0 left) 2024-09-17T16:02:37.510 INFO:tasks.rados.rados.0.smithi181.stdout:335: snap_remove snap 14 2024-09-17T16:02:37.545 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:37.543+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:37.553 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:37.554 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:37.554 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:37.646 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:37.643+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:37.669 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.669 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:37.669 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:37.670 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:37.670 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:37.746 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:37.743+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:37.790 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=29, reqid=client.4560.0:185, at_version=39'11, trim_to=0'0, roll_forward_to=35'9))", 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.900430636999999, 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032500839999999999, 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.791 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887702+0000", 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.792 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.887703+0000", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1399999999999997e-07 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888250+0000", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00054787100000000001 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888279+0000", 2024-09-17T16:02:37.793 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8102999999999998e-05 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888288+0000", 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3819999999999993e-06 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888359+0000", 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0547000000000001e-05 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.794 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.888368+0000", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9010000000000006e-06 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890918+0000", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002549164 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.890952+0000", 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4702000000000001e-05 2024-09-17T16:02:37.795 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s5 39/32 ECSubWrite(tid=30, reqid=client.4560.0:185, at_version=0'0, trim_to=0'0, roll_forward_to=39'11))", 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 59.895878676999999, 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95655153800000003, 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.796 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892254+0000", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892255+0000", 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:37.892270+0000", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5148e-05 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847941+0000", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.95567063299999999 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.847959+0000", 2024-09-17T16:02:37.798 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8009e-05 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848077+0000", 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117796 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848087+0000", 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0328e-05 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.799 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848752+0000", 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00066467800000000004 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:38.848806+0000", 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.41e-05 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.800 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s6 55/53 ECSubWrite(tid=62, reqid=client.4560.0:351, at_version=55'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.415131611, 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003094585, 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.801 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373001+0000", 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.61e-07 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.802 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373230+0000", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022871100000000001 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373269+0000", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8470999999999998e-05 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373278+0000", 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8449999999999998e-06 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.803 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373359+0000", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0401999999999994e-05 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.373368+0000", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1919999999999994e-06 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376052+0000", 2024-09-17T16:02:37.804 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026842319999999999 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.376096+0000", 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3571e-05 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 55/53 ECSubWrite(tid=13, reqid=client.4560.0:366, at_version=55'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 44.391306626999999, 2024-09-17T16:02:37.805 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020885859999999999, 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.806 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.396826+0000", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8999999999999998e-07 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397062+0000", 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000236006 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.807 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397106+0000", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3995000000000001e-05 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397119+0000", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.241e-05 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397201+0000", 2024-09-17T16:02:37.808 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2498999999999997e-05 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.397214+0000", 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2819999999999999e-05 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398870+0000", 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016554829999999999 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.809 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:53.398915+0000", 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5083000000000002e-05 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 64/53 ECSubWrite(tid=81, reqid=osd.14.0:81, at_version=64'38, trim_to=0'0, roll_forward_to=63'36))", 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.445462702, 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038094829999999998, 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.810 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342670+0000", 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342671+0000", 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.342695+0000", 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4366999999999999e-05 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343531+0000", 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00083620199999999997 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.812 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343542+0000", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0672e-05 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343828+0000", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000285834 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.343846+0000", 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8471999999999999e-05 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346440+0000", 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025931499999999998 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346480+0000", 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0064000000000001e-05 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.814 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.442392580000003, 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027362839999999999, 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.815 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345740+0000", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1799999999999999e-07 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0065999999999999e-05 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345845+0000", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4876999999999996e-05 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345856+0000", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1073000000000001e-05 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345935+0000", 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9010000000000004e-05 2024-09-17T16:02:37.817 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345947+0000", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1708999999999999e-05 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348437+0000", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024899560000000002 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348477+0000", 2024-09-17T16:02:37.818 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9375000000000002e-05 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.10s5 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.442351956000003, 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034936559999999999, 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.819 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345781+0000", 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.92e-07 2024-09-17T16:02:37.820 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345815+0000", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3519999999999998e-05 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345868+0000", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3248000000000002e-05 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.345877+0000", 2024-09-17T16:02:37.821 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7970000000000006e-06 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346334+0000", 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045731999999999999 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346345+0000", 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0854e-05 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349226+0000", 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028807569999999998 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349275+0000", 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8968000000000003e-05 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 64/53 ECSubWrite(tid=25, reqid=osd.8.0:25, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:37.823 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.441500683999998, 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002590527, 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.824 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346632+0000", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.73e-07 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346659+0000", 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7055000000000001e-05 2024-09-17T16:02:37.825 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346738+0000", 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8388999999999998e-05 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346746+0000", 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4090000000000003e-06 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.826 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346817+0000", 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0895999999999993e-05 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346826+0000", 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9330000000000007e-06 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349183+0000", 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023570639999999999 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.827 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.349223+0000", 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.9607999999999998e-05 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 35.441470957, 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0037577729999999998, 2024-09-17T16:02:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.829 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346662+0000", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.346683+0000", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0866999999999999e-05 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347481+0000", 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00079794800000000004 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.830 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.347491+0000", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8770000000000005e-06 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348761+0000", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001270188 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.348771+0000", 2024-09-17T16:02:37.831 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.815e-06 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350377+0000", 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016056499999999999 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:02.350420+0000", 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2944000000000001e-05 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.832 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 30.437168523, 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027021179999999999, 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:37.833 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350964+0000", 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350965+0000", 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.99e-07 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.834 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.350989+0000", 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4145999999999999e-05 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351039+0000", 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9858000000000003e-05 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351051+0000", 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2729e-05 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.835 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351123+0000", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1680000000000005e-05 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.351135+0000", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2167000000000001e-05 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353616+0000", 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002481287 2024-09-17T16:02:37.836 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:07.353666+0000", 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0052000000000001e-05 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.342579009000001, 2024-09-17T16:02:37.837 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026861049999999998, 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445554+0000", 2024-09-17T16:02:37.838 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.445555+0000", 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9299999999999997e-07 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446015+0000", 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046028900000000001 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.839 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446034+0000", 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9279e-05 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446041+0000", 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446100+0000", 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8473000000000001e-05 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.840 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446107+0000", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9319999999999999e-06 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448192+0000", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020857570000000001 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448240+0000", 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7469000000000002e-05 2024-09-17T16:02:37.841 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 81/53 ECSubWrite(tid=47, reqid=osd.8.0:45, at_version=0'0, trim_to=0'0, roll_forward_to=81'22))", 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.341253298000002, 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002323982, 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.842 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446880+0000", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446881+0000", 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.232e-06 2024-09-17T16:02:37.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446896+0000", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4976e-05 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446962+0000", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6397999999999997e-05 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446971+0000", 2024-09-17T16:02:37.844 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2609999999999999e-06 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447062+0000", 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0257e-05 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447074+0000", 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2108e-05 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.845 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449155+0000", 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020816559999999999 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449204+0000", 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8093999999999997e-05 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.cs6 81/32 ECSubWrite(tid=102, reqid=osd.0.0:101, at_version=0'0, trim_to=0'0, roll_forward_to=81'49))", 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:37.846 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.341202017000001, 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022743210000000002, 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.847 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446931+0000", 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0900000000000001e-07 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.446942+0000", 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0988999999999999e-05 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.848 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447265+0000", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032216900000000003 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447274+0000", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4800000000000007e-06 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447363+0000", 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9007999999999999e-05 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.849 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.447374+0000", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1121000000000001e-05 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449154+0000", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001779997 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449205+0000", 2024-09-17T16:02:37.850 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0948000000000001e-05 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.19s5 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 18.339280894000002, 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023273109999999999, 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.851 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448852+0000", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.448853+0000", 2024-09-17T16:02:37.852 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1379999999999999e-06 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449276+0000", 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000422757 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449299+0000", 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3098000000000001e-05 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.853 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449308+0000", 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0159999999999997e-06 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449374+0000", 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6029000000000001e-05 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.449385+0000", 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1019000000000001e-05 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.854 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451144+0000", 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001759287 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4966999999999999e-05 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 93/53 ECSubWrite(tid=62, reqid=osd.8.0:62, at_version=93'32, trim_to=0'0, roll_forward_to=84'30))", 2024-09-17T16:02:37.855 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:31.510861+0000", 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.2772714670000003, 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025611649999999998, 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510861+0000", 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.856 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510861+0000", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510862+0000", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4999999999999999e-07 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510885+0000", 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2909999999999999e-05 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.857 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510965+0000", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0501999999999996e-05 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.510973+0000", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1359999999999997e-06 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.511054+0000", 2024-09-17T16:02:37.858 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1085e-05 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.511070+0000", 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6198e-05 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.513384+0000", 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002313954 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.859 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.513423+0000", 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8130000000000003e-05 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 93/53 ECSubWrite(tid=63, reqid=osd.8.0:63, at_version=93'28, trim_to=0'0, roll_forward_to=89'26))", 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:31.512060+0000", 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 6.2760727120000004, 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020781599999999999, 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512060+0000", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512060+0000", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512061+0000", 2024-09-17T16:02:37.861 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9100000000000003e-07 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512083+0000", 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2214000000000001e-05 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512117+0000", 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4366000000000003e-05 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.862 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512126+0000", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9020000000000005e-06 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512183+0000", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7167999999999999e-05 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.512194+0000", 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0696000000000001e-05 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.863 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.514104+0000", 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019101789999999999 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:31.514138+0000", 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3943999999999999e-05 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.864 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 98/53 ECSubWrite(tid=201, reqid=osd.14.0:201, at_version=98'90, trim_to=0'0, roll_forward_to=97'88))", 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.508064+0000", 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.2800688980000001, 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035824810000000002, 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508064+0000", 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.865 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508064+0000", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508065+0000", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.99e-07 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508106+0000", 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1820999999999998e-05 2024-09-17T16:02:37.866 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508147+0000", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.104e-05 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508157+0000", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8789999999999998e-06 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508231+0000", 2024-09-17T16:02:37.867 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3998000000000005e-05 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.508245+0000", 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4110999999999999e-05 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511597+0000", 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003351803 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.868 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511646+0000", 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9129999999999999e-05 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.15s4 98/53 ECSubWrite(tid=68, reqid=osd.8.0:68, at_version=98'34, trim_to=0'0, roll_forward_to=93'32))", 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.511914+0000", 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.276219016, 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003481741, 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.869 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511914+0000", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511914+0000", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.870 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511914+0000", 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1e-07 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511940+0000", 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5911e-05 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512013+0000", 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2713999999999995e-05 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.871 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512023+0000", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0152999999999999e-05 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512100+0000", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7202999999999999e-05 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512112+0000", 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1989e-05 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.872 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515341+0000", 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032294979999999999 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515396+0000", 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4063000000000001e-05 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.873 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.5s4 98/53 ECSubWrite(tid=69, reqid=osd.8.0:69, at_version=98'32, trim_to=0'0, roll_forward_to=97'30))", 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.511943+0000", 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.276189682, 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034864649999999998, 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511943+0000", 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.874 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511943+0000", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511944+0000", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3099999999999998e-07 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.511966+0000", 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2517000000000001e-05 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.875 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512056+0000", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9429e-05 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512063+0000", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9600000000000003e-06 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512149+0000", 2024-09-17T16:02:37.876 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6550000000000003e-05 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.512162+0000", 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2398000000000001e-05 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515403+0000", 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032412209999999999 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.877 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515430+0000", 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6859000000000002e-05 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.0s6 98/53 ECSubWrite(tid=202, reqid=osd.14.0:201, at_version=0'0, trim_to=0'0, roll_forward_to=98'90))", 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:36.513363+0000", 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.274769861, 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026211989999999998, 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:37.878 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513363+0000", 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513363+0000", 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.879 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513363+0000", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7399999999999999e-07 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513374+0000", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0847e-05 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513481+0000", 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000107247 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.880 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513511+0000", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9845e-05 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513705+0000", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000193561 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.513717+0000", 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1798e-05 2024-09-17T16:02:37.881 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515935+0000", 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022180569999999998 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:36.515984+0000", 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9570000000000001e-05 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:37.882 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:37.883 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:37.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:37.885 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:37.882+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:37.985 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:37.982+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:38.032 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:38.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:38.033 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:38.033 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:38.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:38.085 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:38.083+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:38.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:38.148 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:38.148 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:38.149 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-09-17T16:02:38.149 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-09-17T16:02:38.149 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:38.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:38.187 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:38.185+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:38.252 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:38.253 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:21 3.1s0 3:8347559a:::smithi18136759-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.410519081999993, 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014798321, 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.266 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 21 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852756+0000", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852757+0000", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3900000000000004e-07 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852770+0000", 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3409e-05 2024-09-17T16:02:38.268 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852781+0000", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.852789+0000", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0069999999999997e-06 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855856+0000", 2024-09-17T16:02:38.269 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003067223 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.855933+0000", 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6773000000000005e-05 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865810+0000", 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098773300000000001 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.270 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865905+0000", 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5439999999999994e-05 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867554+0000", 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016489180000000001 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:47 3.6s0 3:6d55c691:::smithi18136759-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.397841787999994, 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022622719999999999, 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 47 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.272 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865433+0000", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865434+0000", 2024-09-17T16:02:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8999999999999995e-07 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865441+0000", 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0280000000000003e-06 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865455+0000", 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3862e-05 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.274 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865465+0000", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867296+0000", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001831406 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867372+0000", 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5537000000000005e-05 2024-09-17T16:02:38.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.867795+0000", 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042325499999999999 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868400+0000", 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060551799999999999 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.276 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887964+0000", 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019564088 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888056+0000", 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.1685000000000001e-05 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.277 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:48 3.6s0 3:6d55c691:::smithi18136759-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.397568547000006, 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023961565000000001, 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 48 2024-09-17T16:02:38.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.279 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865707+0000", 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.02e-07 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865713+0000", 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8159999999999999e-06 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865725+0000", 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1809e-05 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.280 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.865734+0000", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6060000000000003e-06 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868049+0000", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023157680000000002 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868100+0000", 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0238000000000002e-05 2024-09-17T16:02:38.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888141+0000", 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020041118 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888234+0000", 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3220000000000005e-05 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.282 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889668+0000", 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001434088 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:50 3.7s0 3:ec2b5851:::smithi18136759-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:38.283 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.394702765000005, 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019246236, 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 50 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.284 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868572+0000", 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868573+0000", 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1699999999999997e-07 2024-09-17T16:02:38.285 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868578+0000", 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5229999999999997e-06 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868594+0000", 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6061000000000001e-05 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.286 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868601+0000", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3289999999999996e-06 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870255+0000", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001653896 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870351+0000", 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6473999999999998e-05 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.287 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.870847+0000", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049566599999999997 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871900+0000", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010525619999999999 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887749+0000", 2024-09-17T16:02:38.288 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015849691999999999 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.887819+0000", 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9215999999999996e-05 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:51 3.7s0 3:ec2b5851:::smithi18136759-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:38.289 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.394650838000004, 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021178519999999999, 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:38.290 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868624+0000", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868625+0000", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.9500000000000003e-07 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868628+0000", 2024-09-17T16:02:38.291 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.608e-06 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868675+0000", 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6727999999999997e-05 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.868684+0000", 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.132e-06 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.292 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871112+0000", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024272569999999999 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.871182+0000", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0059000000000001e-05 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888310+0000", 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017128823000000001 2024-09-17T16:02:38.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.888399+0000", 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8881000000000005e-05 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889803+0000", 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0014035370000000001 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.294 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:74 3.bs0 3:d5fe7fee:::smithi18136759-25:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.374110071999993, 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017195512999999999, 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 74 2024-09-17T16:02:38.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889165+0000", 2024-09-17T16:02:38.296 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889166+0000", 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889179+0000", 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3397999999999999e-05 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.297 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889189+0000", 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4259999999999992e-06 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889212+0000", 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3343999999999999e-05 2024-09-17T16:02:38.298 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891001+0000", 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017887059999999999 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891094+0000", 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3487999999999997e-05 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.299 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.891606+0000", 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00051167900000000002 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892598+0000", 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00099211599999999992 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.300 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906269+0000", 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013670554999999999 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906361+0000", 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2058999999999994e-05 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:75 3.bs0 3:d5fe7fee:::smithi18136759-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.374026897999997, 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018143223, 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.302 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 75 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.303 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889248+0000", 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889249+0000", 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7300000000000004e-07 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889255+0000", 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.507e-06 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889264+0000", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0529999999999996e-06 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.889272+0000", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0479999999999999e-06 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892314+0000", 2024-09-17T16:02:38.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003042664 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.892405+0000", 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0172000000000003e-05 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906710+0000", 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014305356999999999 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906803+0000", 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3467999999999993e-05 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.907391+0000", 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00058798100000000001 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:89 3.6s0 3:6b21f5fe:::smithi18136759-30:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.307 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.359584131000005, 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.073528319999999994, 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 89 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903691+0000", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903692+0000", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.309 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903694+0000", 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6749999999999998e-06 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903701+0000", 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903704+0000", 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.49e-06 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905457+0000", 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0017525290000000001 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905517+0000", 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0226999999999998e-05 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.905822+0000", 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00030490299999999998 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906210+0000", 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00038821299999999999 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977144+0000", 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.070933447999999996 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977219+0000", 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5555000000000004e-05 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:90 3.6s0 3:6b21f5fe:::smithi18136759-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:38.313 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 64.359544440999997, 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.074349779000000005, 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 90 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903731+0000", 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:38.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903733+0000", 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.34e-06 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903739+0000", 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.118e-06 2024-09-17T16:02:38.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.903742+0000", 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4149999999999999e-06 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906009+0000", 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022667709999999999 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.906062+0000", 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3418999999999998e-05 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977459+0000", 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.071396395000000001 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.977514+0000", 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5791000000000002e-05 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:33.978080+0000", 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056612399999999999 2024-09-17T16:02:38.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 44/32 ECSubWriteReply(tid=64, last_complete=44'10, committed=1, applied=1))", 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.324302201000002, 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53213360099999996, 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:38.320 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938973+0000", 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:42.938975+0000", 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.871e-06 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.470990+0000", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.53201507599999998 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471008+0000", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8042000000000001e-05 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471075+0000", 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6439999999999994e-05 2024-09-17T16:02:38.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:43.471107+0000", 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1874e-05 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:388 3.16s0 3:6d55c691:::smithi18136759-16:c [read 0~262144,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.925131352999998, 2024-09-17T16:02:38.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.148899007, 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 388 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338144+0000", 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.63e-07 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338147+0000", 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.129e-06 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338155+0000", 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1389999999999998e-06 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338160+0000", 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1399999999999999e-06 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338288+0000", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128374 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.338295+0000", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7730000000000002e-06 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.342955+0000", 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046598580000000002 2024-09-17T16:02:38.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343006+0000", 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1329000000000003e-05 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487043+0000", 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14403680199999999 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpReadReply(3.16s0 60/53 ECSubReadReply(tid=114, attrs_read=0))", 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.919700644000002, 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14348618299999999, 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343574+0000", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343575+0000", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6400000000000003e-07 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.343629+0000", 2024-09-17T16:02:38.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4472000000000003e-05 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486547+0000", 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.142917403 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486554+0000", 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.503e-06 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486609+0000", 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4527000000000002e-05 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.487061+0000", 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00045181399999999998 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "replica_scrub(pg: 3.15s1,from:0'0,to:56'8,epoch:60/53,start:3:a8000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:38.333 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.776756704999997, 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45217676400000001, 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486518+0000", 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486519+0000", 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9899999999999999e-07 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.486525+0000", 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5949999999999998e-06 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938489+0000", 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.45196369600000003 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938505+0000", 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6064000000000001e-05 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938656+0000", 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015132599999999999 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938666+0000", 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6320000000000003e-06 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:01:58.938695+0000", 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9652e-05 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.337 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:447 3.1s0 3:8347559a:::smithi18136759-7:10 [read 97754~16934,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.922407823, 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014398077, 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 447 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340867+0000", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340869+0000", 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4729999999999999e-06 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340875+0000", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6630000000000004e-06 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340887+0000", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.112e-05 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340896+0000", 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:38.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344260+0000", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0033644320000000001 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344265+0000", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2479999999999998e-06 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353223+0000", 2024-09-17T16:02:38.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089583140000000002 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.353351+0000", 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000128231 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355265+0000", 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019143319999999999 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:448 3.1s0 3:8347559a:::smithi18136759-7:10 [read 0~114688,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.922352216999997, 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015642311999999998, 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 448 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.343 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340923+0000", 2024-09-17T16:02:38.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4799999999999998e-07 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340927+0000", 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.569e-06 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340936+0000", 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9409999999999996e-06 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340942+0000", 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5439999999999998e-06 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344342+0000", 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034007899999999999 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344344+0000", 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.333e-06 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355309+0000", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010965557000000001 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355379+0000", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9503000000000002e-05 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356565+0000", 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011865269999999999 2024-09-17T16:02:38.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:449 3.1s0 3:8347559a:::smithi18136759-7:10 [read 41111~73577,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.922323273000003, 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015941349, 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 449 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340952+0000", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4400000000000001e-07 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340955+0000", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0299999999999998e-06 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340962+0000", 2024-09-17T16:02:38.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.17e-06 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.340968+0000", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.1869999999999996e-06 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344372+0000", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034039690000000002 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.344373+0000", 2024-09-17T16:02:38.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1400000000000001e-06 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355572+0000", 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011199565 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.355632+0000", 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9210999999999999e-05 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:04.356893+0000", 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012616330000000001 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:511 3.6s0 3:60a7d8b5:::smithi18136759-22:head [copy-from ver 12] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 29.888856711999999, 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016414522000000001, 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.353 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 511 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374418+0000", 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374421+0000", 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2189999999999998e-06 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374429+0000", 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4929999999999998e-06 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374449+0000", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0360000000000002e-05 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374459+0000", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0559999999999994e-06 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374533+0000", 2024-09-17T16:02:38.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4171999999999994e-05 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.374651+0000", 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000118495 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.383140+0000", 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084888240000000007 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.384289+0000", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011493639999999999 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390716+0000", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0064265609999999999 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:08.390833+0000", 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000116978 2024-09-17T16:02:38.358 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWrite(3.9s2 74/53 ECSubWrite(tid=138, reqid=osd.14.0:138, at_version=74'39, trim_to=0'0, roll_forward_to=73'35))", 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 25.859339411000001, 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.11895731, 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.359 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403936+0000", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8299999999999997e-07 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.403952+0000", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5957e-05 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522019+0000", 2024-09-17T16:02:38.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.118066429 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522040+0000", 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1769000000000001e-05 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522199+0000", 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015867999999999999 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522215+0000", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5971999999999999e-05 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522867+0000", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065141500000000002 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:12.522893+0000", 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6505000000000001e-05 2024-09-17T16:02:38.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.4560.0:772 3.16s0 3:6d55c691:::smithi18136759-16:head [copy-from ver 39] snapc 21=[21,20,1f,1e,1d,1c,19,18,16,10,c,6] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.790884845000001, 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015349042, 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-09-17T16:02:38.364 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.4560", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 772 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472390+0000", 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472392+0000", 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7349999999999999e-06 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472399+0000", 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0469999999999996e-06 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472416+0000", 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7073999999999999e-05 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472424+0000", 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3580000000000003e-06 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472510+0000", 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5284000000000005e-05 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.472516+0000", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0209999999999996e-06 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.474786+0000", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022700929999999999 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.474869+0000", 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2621999999999996e-05 2024-09-17T16:02:38.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.484369+0000", 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095008660000000002 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.485566+0000", 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011963360000000001 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-09-17T16:02:38.369 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.487506+0000", 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001940322 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.487739+0000", 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000233284 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(3.16s0 87/65 ECSubWriteReply(tid=220, last_complete=87'36, committed=1, applied=1))", 2024-09-17T16:02:38.370 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 12.774488171, 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54478348899999995, 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-09-17T16:02:38.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488787+0000", 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6300000000000001e-07 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:25.488789+0000", 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.567e-06 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033437+0000", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.54464779200000002 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033444+0000", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9099999999999999e-06 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033516+0000", 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2305999999999995e-05 2024-09-17T16:02:38.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-09-17T16:02:26.033570+0000", 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4551e-05 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-09-17T16:02:38.374 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:38.375 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:38.376 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:38.376 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:38.377 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:38.377 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:38.377 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:38.374+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:38.378 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:38.378 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:38.379 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:38.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:38.476 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:38.475+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:38.479 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:38.479 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:38.479 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:38.493 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:38.493 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:38.499 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:38.500 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:38.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:38.510 INFO:tasks.rados.rados.0.smithi181.stdout:335: done (0 left) 2024-09-17T16:02:38.510 INFO:tasks.rados.rados.0.smithi181.stdout:336: read oid 4 snap 35 2024-09-17T16:02:38.510 INFO:tasks.rados.rados.0.smithi181.stdout:336: expect (ObjNum 1475574880 snap 21989 seq_num 1475159936) 2024-09-17T16:02:38.510 INFO:tasks.rados.rados.0.smithi181.stdout:337: copy_from oid 2 from oid 30 current snap is 38 2024-09-17T16:02:38.511 INFO:tasks.rados.rados.0.smithi181.stdout:338: setattr oid 35 current snap is 38 2024-09-17T16:02:38.512 INFO:tasks.rados.rados.0.smithi181.stdout:339: snap_create 2024-09-17T16:02:38.515 INFO:tasks.rados.rados.0.smithi181.stdout:336: expect (ObjNum 1475574880 snap 21989 seq_num 1475159936) 2024-09-17T16:02:38.521 INFO:tasks.rados.rados.0.smithi181.stdout:337: finishing copy_from to smithi18136759-2 2024-09-17T16:02:38.521 INFO:tasks.rados.rados.0.smithi181.stdout:337: got expected ENOENT (src dne) 2024-09-17T16:02:38.521 INFO:tasks.rados.rados.0.smithi181.stdout:337: finishing copy_from racing read to smithi18136759-2 2024-09-17T16:02:38.522 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 35 v 23 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T16:02:38.533 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:38.533 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:38.554 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:38.554 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:38.577 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:38.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:38.577 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:38.575+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:38.597 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:38.597 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:38.615 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:38.615 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:38.619 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:38.619 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:38.624 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:38.640 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:38.640 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:38.646 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:38.643+0000 7fe4398c6640 0 -- 172.21.15.6:0/1320537200 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fe434222090 msgr2=0x7fe434224480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:38.668 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:38.668 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:38.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:38.678 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:38.676+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:38.728 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:38.728 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:38.737 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:38.778 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:38.776+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:38.839 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:38.840 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:38.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:38.879 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:38.877+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:38.951 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:38.951 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:38.961 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:38.980 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:38.978+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:39.064 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.065 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.073 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:39.080 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:39.079+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:39.177 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.177 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.181 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:39.180+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:39.186 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:39.282 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:39.281+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:39.289 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.289 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.297 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:39.383 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:39.381+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:39.400 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.400 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:39.484 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:39.482+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:39.514 INFO:tasks.rados.rados.0.smithi181.stdout:336: done (3 left) 2024-09-17T16:02:39.514 INFO:tasks.rados.rados.0.smithi181.stdout:337: done (2 left) 2024-09-17T16:02:39.514 INFO:tasks.rados.rados.0.smithi181.stdout:338: done (1 left) 2024-09-17T16:02:39.514 INFO:tasks.rados.rados.0.smithi181.stdout:339: done (0 left) 2024-09-17T16:02:39.514 INFO:tasks.rados.rados.0.smithi181.stdout:340: delete oid 29 current snap is 39 2024-09-17T16:02:39.515 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.515 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:340: done (0 left) 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:341: read oid 17 snap -1 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:341: expect (ObjNum 102 snap 24 seq_num 102) 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:342: read oid 21 snap -1 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:342: expect deleted 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:343: rollback oid 15 current snap is 39 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 15 to 20 2024-09-17T16:02:39.521 INFO:tasks.rados.rados.0.smithi181.stdout:344: snap_remove snap 29 2024-09-17T16:02:39.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:02:39.525 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:39.585 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:39.582+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:39.668 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.668 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.677 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:39.687 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:39.683+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:39.780 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.780 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.787 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:39.786+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:39.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:39.888 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:39.886+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:39.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:39.989 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:39.987+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:40.018 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.018 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.027 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:40.089 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:40.087+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:40.129 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.130 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:40.190 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:40.189+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:40.243 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.243 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:40.291 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:40.288+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:40.392 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:40.391+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:40.396 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.396 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:40.494 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:40.492+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:40.507 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.507 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.514 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:02:40.516 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:344: done (3 left) 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:345: read oid 45 snap -1 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:345: expect (ObjNum 90 snap 22 seq_num 90) 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:346: read oid 1 snap 38 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:346: expect (ObjNum 108 snap 26 seq_num 108) 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:347: delete oid 37 current snap is 39 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:343: finishing rollback tid 0 to smithi18136759-15 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:343: finishing rollback tid 1 to smithi18136759-15 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:341: expect (ObjNum 102 snap 24 seq_num 102) 2024-09-17T16:02:40.518 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 49440 2024-09-17T16:02:40.524 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:40.524 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:40.524 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-17T16:02:40.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-09-17T16:02:40.542 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.563 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.584 INFO:tasks.rados.rados.0.smithi181.stdout:343: finishing rollback tid 2 to smithi18136759-15 2024-09-17T16:02:40.584 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 43 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-17T16:02:40.584 INFO:tasks.rados.rados.0.smithi181.stdout:341: done (5 left) 2024-09-17T16:02:40.584 INFO:tasks.rados.rados.0.smithi181.stdout:342: done (4 left) 2024-09-17T16:02:40.584 INFO:tasks.rados.rados.0.smithi181.stdout:343: done (3 left) 2024-09-17T16:02:40.585 INFO:tasks.rados.rados.0.smithi181.stdout:347: done (2 left) 2024-09-17T16:02:40.585 INFO:tasks.rados.rados.0.smithi181.stdout:348: snap_create 2024-09-17T16:02:40.585 INFO:tasks.rados.rados.0.smithi181.stdout:345: expect (ObjNum 90 snap 22 seq_num 90) 2024-09-17T16:02:40.585 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.594 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:40.593+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:40.609 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.630 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.630 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.634 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:40.659 INFO:tasks.rados.rados.0.smithi181.stdout:346: expect (ObjNum 108 snap 26 seq_num 108) 2024-09-17T16:02:40.659 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:40.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:40.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 reset_purged_snaps_last 2024-09-17T16:02:40.703 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1016551 2024-09-17T16:02:40.727 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:40.724+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:40.739 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1296141 2024-09-17T16:02:40.761 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.762 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:40.809 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:40.810 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:40.813 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:40.812+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:40.832 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:40.855 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:40.855 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:40.874 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:40.874 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:40.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:40.890 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:40.890 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:40.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-09-17T16:02:40.897 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:40.895+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:40.914 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:40.937 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:40.937 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:40.959 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:40.960 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:40.960 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:40.981 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:40.982 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:40.994 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:40.994 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:40.995 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:40.995 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:40.998 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:40.996+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:41.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:41.005 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:41.005 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:41.025 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:41.025 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:41.049 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:41.049 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:41.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 reset_purged_snaps_last 2024-09-17T16:02:41.070 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:41.070 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:41.092 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:41.092 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:41.098 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:41.096+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:41.130 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:41.131 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:41.136 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:41.136 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:41.136 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:41.136 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:41.136 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:41.137 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:41.133+0000 7fafaffff640 0 -- 172.21.15.6:0/149353363 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fafb0221d60 msgr2=0x7fafb0224150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:41.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:41.173 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:41.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:41.199 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:41.197+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:41.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-09-17T16:02:41.284 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:41.284 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:41.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:248 3.cs0 3:3808462d:::smithi18136759-6:head [copy-from ver 2] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 58.353241148999999, 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019268496999999999, 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.285 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 248 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928583+0000", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928588+0000", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2220000000000001e-06 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928601+0000", 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2717e-05 2024-09-17T16:02:41.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928608+0000", 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8600000000000004e-06 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928671+0000", 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3791e-05 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928756+0000", 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4293000000000001e-05 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940817+0000", 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01206144 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.942933+0000", 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021155280000000002 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947791+0000", 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048580639999999996 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947850+0000", 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9682999999999998e-05 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:249 3.cs0 3:3808462d:::smithi18136759-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 58.353200037000001, 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013329483, 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.291 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928624+0000", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2699999999999999e-07 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928626+0000", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8940000000000002e-06 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928643+0000", 2024-09-17T16:02:41.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6988999999999999e-05 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928651+0000", 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.267e-06 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928901+0000", 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025058200000000001 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:41.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928955+0000", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3973e-05 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941770+0000", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012814828 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941852+0000", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083000000000007e-05 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941953+0000", 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010034 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:285 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.350017717999997, 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010247877000000001, 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.296 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 285 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931808+0000", 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8779999999999998e-06 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931822+0000", 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3828e-05 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931850+0000", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7500000000000001e-05 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931863+0000", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3738e-05 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931969+0000", 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001052 2024-09-17T16:02:41.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931976+0000", 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938082+0000", 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061057769999999997 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938176+0000", 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3714000000000004e-05 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939069+0000", 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00089279400000000003 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941333+0000", 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022647380000000001 2024-09-17T16:02:41.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941992+0000", 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065853199999999998 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.942053+0000", 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1447999999999998e-05 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:286 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 2] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.349864027999999, 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013358431, 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.303 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 286 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:41.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931960+0000", 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931965+0000", 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9010000000000002e-06 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931974+0000", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2520000000000005e-06 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931981+0000", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5950000000000004e-06 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932071+0000", 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0390999999999996e-05 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932079+0000", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2679999999999994e-06 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939566+0000", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074865030000000003 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939653+0000", 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7336999999999994e-05 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940238+0000", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058448400000000002 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944090+0000", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00385192 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945232+0000", 2024-09-17T16:02:41.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001142784 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945317+0000", 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5080999999999994e-05 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:287 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:41.309 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.349828088999999, 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013571456000000001, 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 287 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:41.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931996+0000", 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931999+0000", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4300000000000002e-06 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932006+0000", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2729999999999997e-06 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932012+0000", 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8089999999999998e-06 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932107+0000", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4751000000000001e-05 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932112+0000", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940481+0000", 2024-09-17T16:02:41.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083682089999999997 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940546+0000", 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5704000000000007e-05 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940923+0000", 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00037686099999999998 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944112+0000", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031884999999999999 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945503+0000", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001391226 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945566+0000", 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3510999999999996e-05 2024-09-17T16:02:41.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 52/32 ECSubWriteReply(tid=51, last_complete=52'22, committed=1, applied=1))", 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.924851936000003, 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.138524809, 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:41.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356972+0000", 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356976+0000", 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2950000000000004e-06 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495052+0000", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13807575 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495065+0000", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3276e-05 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495133+0000", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8274000000000004e-05 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495496+0000", 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036245899999999998 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:344 3.fs0 3:fd9ed431:::smithi18136759-28:head [stat,delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 48.945860007, 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012105121, 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.320 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 344 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335966+0000", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.004e-06 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335978+0000", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1810999999999999e-05 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336003+0000", 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5344e-05 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336017+0000", 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3901e-05 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336125+0000", 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107707 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336136+0000", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1476e-05 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340410+0000", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004273453 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340525+0000", 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115017 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.341450+0000", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00092533800000000005 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.345523+0000", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040730929999999999 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348008+0000", 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024844609999999999 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348068+0000", 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0516000000000001e-05 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=128, reqid=osd.12.0:128, at_version=64'68, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.943924836000001, 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076656809999999997, 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337898+0000", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.337917+0000", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8366999999999999e-05 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339622+0000", 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017049529999999999 2024-09-17T16:02:41.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339629+0000", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3660000000000003e-06 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339905+0000", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027553099999999997 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339915+0000", 2024-09-17T16:02:41.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0124e-05 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345540+0000", 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056253780000000003 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799e-05 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.330 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.942166350000001, 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081855320000000006, 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339692+0000", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4459000000000003e-05 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339726+0000", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4029000000000002e-05 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339733+0000", 2024-09-17T16:02:41.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0790000000000003e-06 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340271+0000", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00053823 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340282+0000", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347801+0000", 2024-09-17T16:02:41.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075190999999999999 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347842+0000", 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1387999999999997e-05 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.10s6 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.936259458000002, 2024-09-17T16:02:41.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011676772, 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.29e-07 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8580000000000001e-05 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346013+0000", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00042032499999999998 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346021+0000", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2269999999999993e-06 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346353+0000", 2024-09-17T16:02:41.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033160900000000002 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346363+0000", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0390999999999999e-05 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357206+0000", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010843148 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357240+0000", 2024-09-17T16:02:41.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4263e-05 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.935352405000003, 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010663848, 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.340 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7099999999999998e-07 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346489+0000", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7731999999999999e-05 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346664+0000", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017518199999999999 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346671+0000", 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1239999999999999e-06 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346721+0000", 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.02e-05 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.55e-06 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357095+0000", 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010364226000000001 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357135+0000", 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9563000000000001e-05 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s5 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.933972728999997, 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092773109999999999, 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347851+0000", 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347861+0000", 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0325e-05 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347893+0000", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1173000000000001e-05 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347900+0000", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.306e-06 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347964+0000", 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3678999999999998e-05 2024-09-17T16:02:41.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347973+0000", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0790000000000007e-06 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357093+0000", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091199190000000006 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357128+0000", 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5055999999999998e-05 2024-09-17T16:02:41.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.932696989, 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090524279999999995, 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349127+0000", 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:41.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349137+0000", 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349186+0000", 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8306999999999997e-05 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349195+0000", 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3640000000000005e-06 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349414+0000", 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021941500000000001 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349424+0000", 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358144+0000", 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087197060000000007 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358179+0000", 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4814999999999998e-05 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.353 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=132, reqid=osd.12.0:132, at_version=64'72, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.930242077999999, 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076285509999999999, 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351581+0000", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351582+0000", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351615+0000", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2962000000000001e-05 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351654+0000", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8847999999999998e-05 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351662+0000", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0099999999999995e-06 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351779+0000", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117569 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.351788+0000", 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5639999999999997e-06 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359157+0000", 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073690580000000004 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.359210+0000", 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2769999999999998e-05 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.357 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 38.929376595999997, 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079374270000000004, 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.358 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352447+0000", 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:41.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352467+0000", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0146999999999999e-05 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352504+0000", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6365000000000002e-05 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352512+0000", 2024-09-17T16:02:41.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352565+0000", 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3613999999999999e-05 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352576+0000", 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1141999999999999e-05 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360356+0000", 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007779584 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360384+0000", 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7847999999999998e-05 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:41.362 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.928402155999997, 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70314915600000005, 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.54e-07 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353470+0000", 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9177e-05 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055585+0000", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70211433599999995 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055600+0000", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4989e-05 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055790+0000", 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019022599999999999 2024-09-17T16:02:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055803+0000", 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3455000000000001e-05 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056540+0000", 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000736289 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056570+0000", 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0530000000000001e-05 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:490 3.fs0 3:fd9ed431:::smithi18136759-28:head [append 0~770048] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.912804416999997, 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007518299, 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:41.367 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 490 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369019+0000", 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369020+0000", 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.322e-06 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369811+0000", 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00079078099999999995 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369823+0000", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.252e-05 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369829+0000", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4140000000000002e-06 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369879+0000", 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0158999999999999e-05 2024-09-17T16:02:41.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.369990+0000", 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111549 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.370711+0000", 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00072055199999999995 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.371929+0000", 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001218217 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376334+0000", 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044050360000000002 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376537+0000", 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020274900000000001 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.372 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:494 3.fs0 3:fd9ed431:::smithi18136759-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.908153558999999, 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082749039999999996, 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 494 2024-09-17T16:02:41.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.51e-07 2024-09-17T16:02:41.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373672+0000", 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2689999999999998e-06 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373678+0000", 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2909999999999998e-06 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373681+0000", 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3730000000000001e-06 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376047+0000", 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002366071 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376106+0000", 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8610000000000003e-05 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381155+0000", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050492460000000003 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381250+0000", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5358000000000004e-05 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381944+0000", 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069423499999999997 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.377 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 81/53 ECSubWrite(tid=74, reqid=osd.15.0:74, at_version=81'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 21.854530329999999, 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092459199999999995, 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:41.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6199999999999999e-07 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427313+0000", 2024-09-17T16:02:41.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0389000000000001e-05 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427799+0000", 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00048568899999999998 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427805+0000", 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1040000000000003e-06 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429581+0000", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017761179999999999 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429592+0000", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0621e-05 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436494+0000", 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069020920000000003 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436539+0000", 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4545000000000001e-05 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s5 85/65 ECSubWrite(tid=214, reqid=client.4560.0:753, at_version=85'24, trim_to=0'0, roll_forward_to=84'20))", 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.819552264999999, 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16302681799999999, 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.382 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462272+0000", 2024-09-17T16:02:41.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462292+0000", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0965999999999999e-05 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624643+0000", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.162350088 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624650+0000", 2024-09-17T16:02:41.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4290000000000004e-06 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624728+0000", 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7996000000000003e-05 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624749+0000", 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0990000000000001e-05 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625270+0000", 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052091300000000002 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625298+0000", 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7733999999999999e-05 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.386 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:41.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:41.389 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:41.386+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:41.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-09-17T16:02:41.488 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:41.487+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:345: done (2 left) 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:346: done (1 left) 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:348: done (0 left) 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:349: rollback oid 15 current snap is 40 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 15 to 20 2024-09-17T16:02:41.520 INFO:tasks.rados.rados.0.smithi181.stdout:350: setattr oid 21 current snap is 40 2024-09-17T16:02:41.522 INFO:tasks.rados.rados.0.smithi181.stdout:351: snap_remove snap 39 2024-09-17T16:02:41.547 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:41.548 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:41.548 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:41.548 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:41.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:41.589 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:41.587+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:41.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-09-17T16:02:41.666 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:41.666 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:41.666 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:41.666 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:41.667 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:41.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:41.690 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:41.688+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:41.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 reset_purged_snaps_last 2024-09-17T16:02:41.791 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:41.789+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:41.829 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:41.829 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=182, reqid=osd.11.0:182, at_version=46'20, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.908227764999999, 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066826129999999996, 2024-09-17T16:02:41.830 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5800000000000001e-07 2024-09-17T16:02:41.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917886+0000", 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6931999999999999e-05 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918192+0000", 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030561400000000001 2024-09-17T16:02:41.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918355+0000", 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016328600000000001 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918431+0000", 2024-09-17T16:02:41.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5690000000000002e-05 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918441+0000", 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9450000000000005e-06 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924513+0000", 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060725709999999997 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924542+0000", 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8317e-05 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.836 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.907548822000003, 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060611290000000002, 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.837 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.96e-07 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0068000000000001e-05 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010144 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918678+0000", 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0900000000000005e-06 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918750+0000", 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2100000000000004e-05 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918761+0000", 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0767999999999999e-05 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924554+0000", 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057936560000000003 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924599+0000", 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4710999999999999e-05 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:41.843 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.907511722000002, 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059181870000000001, 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.845 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918576+0000", 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918591+0000", 2024-09-17T16:02:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5583999999999999e-05 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918817+0000", 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022566799999999999 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918825+0000", 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9389999999999997e-06 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918894+0000", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9332000000000006e-05 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918904+0000", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9860000000000006e-06 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924448+0000", 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055442470000000004 2024-09-17T16:02:41.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924493+0000", 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4947000000000002e-05 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.906539111999997, 2024-09-17T16:02:41.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056807560000000003, 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919549+0000", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919571+0000", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2155999999999999e-05 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919754+0000", 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018332299999999999 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919767+0000", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2656e-05 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920277+0000", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051049799999999996 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920289+0000", 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1388e-05 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925184+0000", 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048955539999999999 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925228+0000", 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4202000000000001e-05 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.906499328000002, 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056906200000000004, 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1299999999999999e-07 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919607+0000", 2024-09-17T16:02:41.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9062000000000001e-05 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919796+0000", 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018866 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919803+0000", 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4150000000000002e-06 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920325+0000", 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052165800000000004 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920334+0000", 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925242+0000", 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049072009999999999 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925278+0000", 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6634000000000002e-05 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 49.467325207999998, 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067241940000000002, 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.858 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358763+0000", 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1710000000000001e-06 2024-09-17T16:02:41.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358778+0000", 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5158e-05 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358817+0000", 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8832999999999999e-05 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358826+0000", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2390000000000008e-06 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358909+0000", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2556e-05 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358920+0000", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0910000000000001e-05 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365448+0000", 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006528748 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365486+0000", 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7579000000000001e-05 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/53 ECSubWrite(tid=23, reqid=client.4560.0:346, at_version=54'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 49.466590238999999, 2024-09-17T16:02:41.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059831219999999996, 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359521+0000", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3705000000000001e-05 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359561+0000", 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9879999999999998e-05 2024-09-17T16:02:41.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359570+0000", 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1640000000000007e-06 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359645+0000", 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4765000000000007e-05 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359656+0000", 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0702e-05 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365440+0000", 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057839320000000003 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365480+0000", 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0219000000000002e-05 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 69/32 ECSubWrite(tid=70, reqid=osd.2.0:70, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.487242889999997, 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24156233099999999, 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:41.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338868+0000", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3495000000000001e-05 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579761+0000", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24089318000000001 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579766+0000", 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579834+0000", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7265000000000001e-05 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579841+0000", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.029e-06 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580381+0000", 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000540626 2024-09-17T16:02:41.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580406+0000", 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4958999999999999e-05 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s5 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.472746323999999, 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32307851999999998, 2024-09-17T16:02:41.871 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353366+0000", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5431000000000001e-05 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675724+0000", 2024-09-17T16:02:41.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32235754599999999 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675729+0000", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1100000000000002e-06 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675814+0000", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.551e-05 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675822+0000", 2024-09-17T16:02:41.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676394+0000", 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057224999999999995 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676419+0000", 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5049e-05 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.875 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.394036065000002, 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068757990000000001, 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:41.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432072+0000", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0971e-05 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434134+0000", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002062342 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434144+0000", 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:41.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434203+0000", 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8757e-05 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434214+0000", 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0879e-05 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438888+0000", 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046737510000000003 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438927+0000", 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8779000000000003e-05 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.880 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.393143041999998, 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059654479999999999, 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432945+0000", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432987+0000", 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2401999999999998e-05 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433023+0000", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5562999999999998e-05 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433033+0000", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0539e-05 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433334+0000", 2024-09-17T16:02:41.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000300952 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433343+0000", 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.302e-06 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438871+0000", 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055279500000000002 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438909+0000", 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7951999999999998e-05 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=94, reqid=osd.2.0:94, at_version=79'78, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.391990724999999, 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079659940000000005, 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.885 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434097+0000", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434118+0000", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0834000000000001e-05 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440520+0000", 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064023750000000001 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440528+0000", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6120000000000001e-06 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440597+0000", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9429000000000002e-05 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440608+0000", 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1199000000000001e-05 2024-09-17T16:02:41.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442024+0000", 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001415166 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442062+0000", 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8469e-05 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=98, reqid=client.4560.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.378231733, 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25686310400000001, 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447856+0000", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.71e-07 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447869+0000", 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2795e-05 2024-09-17T16:02:41.891 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703761+0000", 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25589224300000002 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703768+0000", 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6599999999999998e-06 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.892 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703842+0000", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4402999999999999e-05 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703857+0000", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4737e-05 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704682+0000", 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082487599999999999 2024-09-17T16:02:41.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704718+0000", 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6519000000000001e-05 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.396078214999999, 2024-09-17T16:02:41.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066388380000000002, 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0100000000000001e-07 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430030+0000", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1211000000000001e-05 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430055+0000", 2024-09-17T16:02:41.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5411000000000001e-05 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430066+0000", 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0635000000000001e-05 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430142+0000", 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5414999999999995e-05 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430154+0000", 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2877000000000001e-05 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436610+0000", 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064551849999999996 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436647+0000", 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7902999999999999e-05 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.898 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.395078035000001, 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056630860000000003, 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431010+0000", 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3399999999999997e-07 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431035+0000", 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5491999999999999e-05 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431082+0000", 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6853e-05 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431092+0000", 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0164e-05 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431183+0000", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1129000000000001e-05 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431195+0000", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1191e-05 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436653+0000", 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054587730000000001 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436672+0000", 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.855e-05 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.19s4 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.377493961999999, 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063649290000000001, 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.903 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448594+0000", 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448861+0000", 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026757000000000001 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448891+0000", 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9943000000000001e-05 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448897+0000", 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.823e-06 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449029+0000", 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013207599999999999 2024-09-17T16:02:41.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449043+0000", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.414e-05 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454916+0000", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058721440000000001 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454958+0000", 2024-09-17T16:02:41.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2295000000000001e-05 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 81/53 ECSubWrite(tid=155, reqid=osd.14.0:155, at_version=81'47, trim_to=0'0, roll_forward_to=77'43))", 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.375074082000001, 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065741319999999999, 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451014+0000", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451051+0000", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7224999999999997e-05 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451087+0000", 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5895000000000001e-05 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451097+0000", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451168+0000", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0992000000000001e-05 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0185e-05 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.911 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457549+0000", 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063700420000000002 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457587+0000", 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8297000000000003e-05 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.912 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 93/32 ECSubWrite(tid=133, reqid=osd.2.0:133, at_version=93'117, trim_to=0'0, roll_forward_to=92'115))", 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.319432537000001, 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059325740000000004, 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.913 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506655+0000", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.16e-07 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506677+0000", 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2242000000000002e-05 2024-09-17T16:02:41.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506717+0000", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0519999999999998e-05 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506726+0000", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3329999999999997e-06 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506996+0000", 2024-09-17T16:02:41.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027085500000000002 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507008+0000", 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1080999999999999e-05 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512549+0000", 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055410959999999997 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512587+0000", 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8231e-05 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 93/53 ECSubWrite(tid=89, reqid=osd.15.0:89, at_version=93'41, trim_to=0'0, roll_forward_to=84'39))", 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.318212811, 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076683949999999997, 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.917 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:41.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.49e-07 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507894+0000", 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9981000000000001e-05 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.919 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507975+0000", 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1248999999999994e-05 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507984+0000", 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508051+0000", 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6963000000000002e-05 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508062+0000", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1418e-05 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515497+0000", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074346079999999997 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515542+0000", 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5818000000000001e-05 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.921 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 98/53 ECSubWrite(tid=92, reqid=osd.15.0:91, at_version=0'0, trim_to=0'0, roll_forward_to=98'43))", 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.3074377860000004, 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00573932, 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:41.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518650+0000", 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6300000000000005e-07 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518670+0000", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0757000000000001e-05 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518703+0000", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2963999999999998e-05 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518712+0000", 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4710000000000007e-06 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518782+0000", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0405000000000001e-05 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518793+0000", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.045e-05 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.524353+0000", 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055602189999999999 2024-09-17T16:02:41.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.524388+0000", 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5391e-05 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:41.926 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:41.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:41.929 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:41.929 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-09-17T16:02:41.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 reset_purged_snaps_last 2024-09-17T16:02:41.930 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:41.925+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:42.028 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:42.026+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:42.077 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:42.077 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:42.077 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:42.077 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:42.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:42.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 reset_purged_snaps_last 2024-09-17T16:02:42.129 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:42.128+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:42.195 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:42.195 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:42.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:42.230 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:42.228+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:42.331 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:42.329+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:42.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 reset_purged_snaps_last 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.373 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.502609374000002, 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019947230999999999, 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867809+0000", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867810+0000", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2399999999999997e-07 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867830+0000", 2024-09-17T16:02:42.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9922999999999999e-05 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881759+0000", 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01392914 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881768+0000", 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7809999999999994e-06 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881836+0000", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8491999999999994e-05 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881849+0000", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2418e-05 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887709+0000", 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058604850000000003 2024-09-17T16:02:42.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887756+0000", 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7168e-05 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=5, reqid=client.4560.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.494564914999998, 2024-09-17T16:02:42.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012978303, 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875854+0000", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3799999999999999e-07 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875860+0000", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.02e-06 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875972+0000", 2024-09-17T16:02:42.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011225100000000001 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875986+0000", 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3526000000000001e-05 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877064+0000", 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00107855 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877074+0000", 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888801+0000", 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011726927999999999 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888832+0000", 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0750999999999997e-05 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=6, reqid=client.4560.0:52, at_version=35'7, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.494557708000002, 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012993864000000001, 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875861+0000", 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3199999999999999e-07 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875913+0000", 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1820999999999997e-05 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876038+0000", 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012561899999999999 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876048+0000", 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.166e-06 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877256+0000", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012082309999999999 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.877264+0000", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0399999999999993e-06 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888835+0000", 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011570815 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888855+0000", 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0040000000000001e-05 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 35/32 ECSubWrite(tid=7, reqid=client.4560.0:53, at_version=35'8, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.494504311, 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012966596, 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.387 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875914+0000", 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.86e-07 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.875921+0000", 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1700000000000002e-06 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876450+0000", 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052973099999999997 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876457+0000", 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1029999999999996e-06 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879593+0000", 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003135597 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.879606+0000", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3142000000000001e-05 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888861+0000", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092553359999999994 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888881+0000", 2024-09-17T16:02:42.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9330999999999998e-05 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=12, reqid=client.4560.0:64, at_version=35'3, trim_to=0'0, roll_forward_to=35'1))", 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.488927258999993, 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069303020000000002, 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.392 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881491+0000", 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881492+0000", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1060000000000001e-06 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.881730+0000", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00023725499999999999 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882445+0000", 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071539399999999995 2024-09-17T16:02:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882453+0000", 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2830000000000001e-06 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882512+0000", 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8569000000000001e-05 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882521+0000", 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1290000000000002e-06 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888395+0000", 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058743440000000001 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888422+0000", 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6222000000000001e-05 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.396 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:61 3.5s0 3:a2653429:::smithi18136759-21:head [append 0~344064,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.487575406000005, 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075767569999999999, 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:42.397 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 61 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882843+0000", 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.882845+0000", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5650000000000001e-06 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883323+0000", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047814299999999999 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883338+0000", 2024-09-17T16:02:42.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.504e-05 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883346+0000", 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0760000000000003e-06 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5992999999999997e-05 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883498+0000", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6426000000000001e-05 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.884493+0000", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00099483199999999996 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889296+0000", 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048030460000000001 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890335+0000", 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001038546 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890420+0000", 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.509e-05 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.402 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:62 3.5s0 3:a2653429:::smithi18136759-21:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.487053571000004, 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007574038, 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 62 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883365+0000", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883366+0000", 2024-09-17T16:02:42.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1800000000000005e-07 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883374+0000", 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1920000000000005e-06 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883384+0000", 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0409e-05 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883390+0000", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5650000000000001e-06 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887851+0000", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044615319999999998 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.887944+0000", 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2359000000000001e-05 2024-09-17T16:02:42.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888411+0000", 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00046751900000000001 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889298+0000", 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088712999999999997 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890854+0000", 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001555753 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890939+0000", 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4860999999999999e-05 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.408 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:63 3.5s0 3:a2653429:::smithi18136759-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.487016492999999, 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010045017999999999, 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 63 2024-09-17T16:02:42.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883402+0000", 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883403+0000", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6300000000000005e-07 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883406+0000", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2320000000000001e-06 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883414+0000", 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9899999999999997e-06 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.883419+0000", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5199999999999997e-06 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.888936+0000", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055166759999999999 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.889040+0000", 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010429699999999999 2024-09-17T16:02:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891231+0000", 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021904110000000002 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.891322+0000", 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0989000000000006e-05 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.893447+0000", 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021253399999999999 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s3 35/32 ECSubWrite(tid=18, reqid=client.4560.0:65, at_version=0'0, trim_to=0'0, roll_forward_to=35'4))", 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.479743467000006, 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080117299999999999, 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.414 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890675+0000", 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890676+0000", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3499999999999996e-07 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.890683+0000", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1130000000000003e-06 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892226+0000", 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015435869999999999 2024-09-17T16:02:42.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892230+0000", 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9360000000000001e-06 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892272+0000", 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1822e-05 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892278+0000", 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2210000000000002e-06 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898638+0000", 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063600380000000001 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.898687+0000", 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8477999999999999e-05 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.418 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=4, last_complete=35'2, committed=1, applied=1))", 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 68.478203894999993, 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013072064, 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.419 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892215+0000", 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.892217+0000", 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.063e-06 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905154+0000", 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012937357999999999 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905163+0000", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.636e-06 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905248+0000", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5352000000000005e-05 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905287+0000", 2024-09-17T16:02:42.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8356999999999999e-05 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 47/32 ECSubWrite(tid=57, reqid=client.4560.0:282, at_version=47'18, trim_to=0'0, roll_forward_to=46'16))", 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.436343874999999, 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069921009999999997, 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934075+0000", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934076+0000", 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6700000000000002e-07 2024-09-17T16:02:42.424 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934104+0000", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8330000000000002e-05 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934155+0000", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1490999999999999e-05 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934165+0000", 2024-09-17T16:02:42.425 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0121e-05 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934248+0000", 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083999999999995e-05 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.934258+0000", 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0808999999999999e-05 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.426 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941026+0000", 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006767421 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941067+0000", 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0978e-05 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 47/32 ECSubWrite(tid=59, reqid=client.4560.0:280, at_version=47'23, trim_to=0'0, roll_forward_to=46'19))", 2024-09-17T16:02:42.427 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.433857949999997, 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074766839999999999, 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.428 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936561+0000", 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936562+0000", 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.64e-07 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936585+0000", 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3082e-05 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.429 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936627+0000", 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2141999999999997e-05 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.936636+0000", 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1509999999999993e-06 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.430 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937201+0000", 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00056518400000000004 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.937210+0000", 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1050000000000001e-06 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.431 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944001+0000", 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067911480000000003 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944037+0000", 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5908000000000002e-05 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 54/32 ECSubWrite(tid=95, reqid=client.4560.0:345, at_version=54'43, trim_to=0'0, roll_forward_to=52'41))", 2024-09-17T16:02:42.432 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.020172352000003, 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007077552, 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.433 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350246+0000", 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9799999999999999e-07 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350272+0000", 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5619e-05 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.434 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350409+0000", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013671899999999999 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350418+0000", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4509999999999998e-06 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350893+0000", 2024-09-17T16:02:42.435 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047499899999999998 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.350904+0000", 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0977e-05 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357278+0000", 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063733080000000003 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.436 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.357324+0000", 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6180999999999997e-05 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 64/53 ECSubWriteReply(tid=28, last_complete=64'12, committed=1, applied=1))", 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 40.009461813000001, 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79147448499999995, 2024-09-17T16:02:42.437 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.438 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360957+0000", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.75e-07 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360961+0000", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9550000000000002e-06 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152290+0000", 2024-09-17T16:02:42.439 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79132856699999998 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152299+0000", 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1980000000000007e-06 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152377+0000", 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8317000000000002e-05 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.440 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:03.152431+0000", 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3773000000000002e-05 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.441 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 70/32 ECSubWrite(tid=171, reqid=client.4560.0:509, at_version=70'105, trim_to=0'0, roll_forward_to=69'101))", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.992860690000001, 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14998041000000001, 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.442 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377558+0000", 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377559+0000", 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7800000000000001e-07 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377579+0000", 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0820999999999999e-05 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.443 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526840+0000", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14926072600000001 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526849+0000", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2499999999999995e-06 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526946+0000", 2024-09-17T16:02:42.444 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7052000000000005e-05 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.526954+0000", 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1940000000000001e-06 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.527499+0000", 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054546100000000001 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.445 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.527538+0000", 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9227999999999998e-05 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s1 81/53 ECSubWrite(tid=76, reqid=osd.15.0:74, at_version=0'0, trim_to=0'0, roll_forward_to=81'33))", 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 22.923501310999999, 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082149000000000007, 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.446 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446917+0000", 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.447 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446918+0000", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4199999999999995e-07 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446930+0000", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1639e-05 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446971+0000", 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1591000000000002e-05 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.448 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.446979+0000", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0360000000000006e-06 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.447063+0000", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3627000000000004e-05 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.447073+0000", 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0482000000000001e-05 2024-09-17T16:02:42.449 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455094+0000", 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080202699999999995 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455132+0000", 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8513000000000002e-05 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.450 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.as6 86/65 ECSubWrite(tid=287, reqid=client.4560.0:768, at_version=86'56, trim_to=0'0, roll_forward_to=86'53))", 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.877289299000001, 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79617869100000005, 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.451 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493129+0000", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493130+0000", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493150+0000", 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0514999999999999e-05 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.452 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288281+0000", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.795130106 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288297+0000", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6243e-05 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288454+0000", 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015702500000000001 2024-09-17T16:02:42.453 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.288488+0000", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4517000000000002e-05 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289268+0000", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000779829 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289308+0000", 2024-09-17T16:02:42.454 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9774999999999998e-05 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 93/32 ECSubWrite(tid=292, reqid=osd.12.0:291, at_version=0'0, trim_to=0'0, roll_forward_to=93'205))", 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.865145682, 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.008291484, 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.455 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505273+0000", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505274+0000", 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2900000000000002e-07 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.456 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505285+0000", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1688000000000001e-05 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505321+0000", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5126000000000002e-05 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505329+0000", 2024-09-17T16:02:42.457 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1340000000000004e-06 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505453+0000", 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012396 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.505464+0000", 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1223000000000001e-05 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.458 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513528+0000", 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080639379999999997 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513564+0000", 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6486000000000003e-05 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds4 93/32 ECSubWrite(tid=293, reqid=osd.12.0:293, at_version=93'207, trim_to=0'0, roll_forward_to=93'205))", 2024-09-17T16:02:42.459 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 10.864232391, 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074650159999999997, 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506186+0000", 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506187+0000", 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.23e-07 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506215+0000", 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7881e-05 2024-09-17T16:02:42.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506258+0000", 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3362999999999998e-05 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506266+0000", 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3829999999999992e-06 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507457+0000", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001191012 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507468+0000", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0492999999999999e-05 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513624+0000", 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061560909999999998 2024-09-17T16:02:42.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.513651+0000", 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.707e-05 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:852 3.5s0 3:a2653429:::smithi18136759-21:head [create,setxattr WdO^pQXrObdoVqtkUig`ciPYXeq\\sPonbWU_iecS (1647) in=40b,setxattr Ue\\cXlYSk`ngpaiWk]Titth_O_oefh]TRhVWdrf[ (1719) in=40b,setxattr Qss^SV\\\\lNqrslOO^^kUhpf_ceRXZRe_da[]g[Y_ (1786) in=40b,setxattr ffPV (1331) in=4b,setxattr oRoPidfefrbi[aiXhTl^_dmXlPZX_rjkRiWO`sXr (7928) in=40b,setxattr XYhNZhe^_fVoNOQVTX\\bOg[lXabQeRasj_gP_Scg (735) in=40b,setxattr YtsSZVZRrXkm]aaYOOnbaVjOZbNRSc_OqXahdX\\a (1333) in=40b,setxattr SPQcXVV\\tac_nnWT^mN_ackrnVads``qljnec[W^ (11996) in=40b,setxattr pUQgitllb^egXpb`gNdZ\\T_YiVXXgo`j\\r[ZrcQf (1717) in=40b,setxattr _Oon]ftjUpR[cXXrpehdNmefTUlOQY\\VRbmbRbat (1785) in=40b,setxattr cqdloqNeRjaplTadmOndod_at[^aUtXajQhjrmd[ (1857) in=40b,setxattr soY_SafOai[jUTmsgtdqYj]UQpUOVtPsTbQQrink (10596) in=40b,setxattr \\TajsXp]pWXSR`kQPVkid[aelZNqc[Vhk_YShla] (1330) in=40b,setxattr rU^f_NQYY (1331) in=9b,setxattr YOYYlgc^URehmPobNWkZcOimYcoTsikOarojgZSd (1401) in=40b,setxattr kaPXtsiiYOWqq^WPYltiW]VSUnNPNY__khtbUbRj (1402) in=40b,setxattr Z_keq\\cb`]tQaZi[kkQYm`nag[`[RVpm_[`Ng^_k (629) in=40b,setxattr oOXZYQrmikb^maPdbcfbW_ofpNPbg[XN_f_jeQnZ (13964) in=40b,setxattr `XUhhTYcQNbeiToNmdXbaXmYfrfQ`OYjceUpramV (19224) in=40b,setxattr tVWXbUjiO`rYiNVdkqYamtZfdNT`kO]a^mTkg\\UR (1225) in=40b] snapc 29=[29,28,26,25,22,21,1c,19,18,16,c,6] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:41.521303+0000", 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.84911510400000001, 2024-09-17T16:02:42.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095235100000000007, 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 852 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521303+0000", 2024-09-17T16:02:42.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521303+0000", 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521305+0000", 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8640000000000001e-06 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521437+0000", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013207299999999999 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521459+0000", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1911000000000001e-05 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521471+0000", 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1787000000000001e-05 2024-09-17T16:02:42.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521524+0000", 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2762000000000001e-05 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.521529+0000", 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1170000000000003e-06 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:42.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.524310+0000", 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002781015 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.524447+0000", 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013746099999999999 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.525525+0000", 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0010772539999999999 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.526705+0000", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011808120000000001 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.530717+0000", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004011382 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.530827+0000", 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110072 2024-09-17T16:02:42.470 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.471 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.471 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.471 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:42.471 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:42.471 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:42.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:42.473 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:42.469+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:42.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 reset_purged_snaps_last 2024-09-17T16:02:42.521 INFO:tasks.rados.rados.0.smithi181.stdout:351: done (2 left) 2024-09-17T16:02:42.521 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 36 current snap is 40 2024-09-17T16:02:42.521 INFO:tasks.rados.rados.0.smithi181.stdout:349: finishing rollback tid 0 to smithi18136759-15 2024-09-17T16:02:42.521 INFO:tasks.rados.rados.0.smithi181.stdout:352: seq_num 133 ranges {409600=688128,1097728=720896,1818624=475136,2293760=360448} 2024-09-17T16:02:42.526 INFO:tasks.rados.rados.0.smithi181.stdout:352: writing smithi18136759-36 from 409600 to 1097728 tid 1 2024-09-17T16:02:42.531 INFO:tasks.rados.rados.0.smithi181.stdout:352: writing smithi18136759-36 from 1097728 to 1818624 tid 2 2024-09-17T16:02:42.533 INFO:tasks.rados.rados.0.smithi181.stdout:352: writing smithi18136759-36 from 1818624 to 2293760 tid 3 2024-09-17T16:02:42.536 INFO:tasks.rados.rados.0.smithi181.stdout:352: writing smithi18136759-36 from 2293760 to 2654208 tid 4 2024-09-17T16:02:42.536 INFO:tasks.rados.rados.0.smithi181.stdout:353: rollback oid 4 current snap is 40 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 4 to 23 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:349: finishing rollback tid 1 to smithi18136759-15 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:349: finishing rollback tid 2 to smithi18136759-15 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 47 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 21 v 39 (ObjNum 33 snap 0 seq_num 1475010240) dirty exists 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 1 to smithi18136759-36 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:349: done (3 left) 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:350: done (2 left) 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:354: read oid 1 snap -1 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:354: expect (ObjNum 108 snap 26 seq_num 108) 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:355: copy_from oid 31 from oid 3 current snap is 40 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 44 current snap is 40 2024-09-17T16:02:42.537 INFO:tasks.rados.rados.0.smithi181.stdout:356: seq_num 134 ranges {5324800=704512,6029312=475136,6504448=573440,7077888=507904} 2024-09-17T16:02:42.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:02:42.569 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18136759-44 from 5324800 to 6029312 tid 1 2024-09-17T16:02:42.572 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18136759-44 from 6029312 to 6504448 tid 2 2024-09-17T16:02:42.574 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18136759-44 from 6504448 to 7077888 tid 3 2024-09-17T16:02:42.578 INFO:tasks.rados.rados.0.smithi181.stdout:356: writing smithi18136759-44 from 7077888 to 7585792 tid 4 2024-09-17T16:02:42.578 INFO:tasks.rados.rados.0.smithi181.stdout:357: snap_remove snap 31 2024-09-17T16:02:42.610 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:42.607+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:42.642 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:42.643 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:42.643 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:42.643 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:42.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:42.673 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:42.671+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:42.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 reset_purged_snaps_last 2024-09-17T16:02:42.774 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:42.771+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:42.838 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:42.839 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:42.839 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:42.839 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:42.839 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:42.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:42.874 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:42.873+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:42.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 reset_purged_snaps_last 2024-09-17T16:02:42.975 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:42.974+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:24 3.ds0 3:be8168ea:::smithi18136759-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.172512363999999, 2024-09-17T16:02:43.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.04871255, 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 24 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856435+0000", 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6700000000000003e-07 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856438+0000", 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0089999999999999e-06 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856446+0000", 2024-09-17T16:02:43.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1240000000000005e-06 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856452+0000", 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8490000000000002e-06 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858431+0000", 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019785380000000002 2024-09-17T16:02:43.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858505+0000", 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4269000000000003e-05 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876493+0000", 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017988067999999999 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.876576+0000", 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2981999999999999e-05 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905147+0000", 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.028571144 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:28 3.ds0 3:b6a72c22:::smithi18136759-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.172485543999997, 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043697429000000003, 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 28 2024-09-17T16:02:43.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856461+0000", 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856462+0000", 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2599999999999998e-07 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856586+0000", 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012445300000000001 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856602+0000", 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5877000000000001e-05 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856609+0000", 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5869999999999998e-06 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858561+0000", 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019522739999999999 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.858680+0000", 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118678 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859415+0000", 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073567700000000001 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864061+0000", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046461560000000002 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900068+0000", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036006330000000003 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900159+0000", 2024-09-17T16:02:43.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0970999999999993e-05 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:29 3.ds0 3:b6a72c22:::smithi18136759-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.172326921000007, 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043754779000000001, 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.045 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 29 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856620+0000", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856621+0000", 2024-09-17T16:02:43.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2099999999999996e-07 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856624+0000", 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7000000000000002e-06 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856631+0000", 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9249999999999998e-06 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856638+0000", 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859904+0000", 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032664679999999998 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.859980+0000", 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5711999999999996e-05 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860364+0000", 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038396600000000002 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864272+0000", 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003907888 2024-09-17T16:02:43.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900273+0000", 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036001247 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900375+0000", 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101535 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.051 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:30 3.ds0 3:b6a72c22:::smithi18136759-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.172301094999995, 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.048780709999999998, 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.052 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 30 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856646+0000", 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4499999999999997e-07 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856657+0000", 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856664+0000", 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.148e-06 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.856673+0000", 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.098e-06 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860632+0000", 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039593440000000001 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.860703+0000", 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0197000000000006e-05 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900473+0000", 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039770158 2024-09-17T16:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.900637+0000", 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000164356 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905426+0000", 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047893579999999996 2024-09-17T16:02:43.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:40 3.ds0 3:b22f7774:::smithi18136759-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.167297742000002, 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039805350000000003, 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.058 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 40 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861649+0000", 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861651+0000", 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.525e-06 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861832+0000", 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018106599999999999 2024-09-17T16:02:43.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861846+0000", 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4497e-05 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861854+0000", 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0390000000000005e-06 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.791e-05 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862054+0000", 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014139799999999999 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.862812+0000", 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000758028 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867004+0000", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004192594 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901412+0000", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034408162999999999 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901454+0000", 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2129999999999998e-05 2024-09-17T16:02:43.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:41 3.ds0 3:b22f7774:::smithi18136759-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.167071436000001, 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039711297, 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.064 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 41 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861875+0000", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861876+0000", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4e-07 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861880+0000", 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6160000000000002e-06 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861887+0000", 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0890000000000002e-06 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861893+0000", 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8649999999999998e-06 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863320+0000", 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001427212 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863412+0000", 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1540999999999997e-05 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.863845+0000", 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043304900000000002 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.867006+0000", 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031615369999999999 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901546+0000", 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034539746000000003 2024-09-17T16:02:43.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901587+0000", 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0802e-05 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:42 3.ds0 3:b22f7774:::smithi18136759-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:43.070 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.167038942000005, 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.043739746000000003, 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 42 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861908+0000", 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861909+0000", 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9400000000000005e-07 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861912+0000", 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3019999999999999e-06 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861919+0000", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3549999999999999e-06 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.861925+0000", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8529999999999997e-06 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864137+0000", 2024-09-17T16:02:43.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022118649999999999 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.864205+0000", 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8396e-05 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901627+0000", 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037421487000000003 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.075 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.901674+0000", 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7577000000000002e-05 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905648+0000", 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039733169999999996 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:54 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.158144133999997, 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035090831000000003, 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.077 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 54 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870803+0000", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9800000000000004e-07 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870806+0000", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3549999999999999e-06 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870813+0000", 2024-09-17T16:02:43.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1539999999999996e-06 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.870819+0000", 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0700000000000003e-06 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873071+0000", 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022519200000000001 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.873154+0000", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3746999999999996e-05 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902680+0000", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.029525513 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.902767+0000", 2024-09-17T16:02:43.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6837999999999998e-05 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:33.905894+0000", 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031267360000000002 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 46/32 ECSubWriteReply(tid=54, last_complete=46'16, committed=1, applied=1))", 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:43.082 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 58.090097919000002, 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62914934300000003, 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938849+0000", 2024-09-17T16:02:43.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938850+0000", 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.06e-07 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.938855+0000", 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0010000000000001e-06 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567797+0000", 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62894212500000002 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567814+0000", 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6847e-05 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567931+0000", 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011766500000000001 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.567998+0000", 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6999e-05 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:284 3.as0 3:5583c93f:::smithi18136759-40:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.095428941000002, 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.386964417, 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 284 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933518+0000", 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933519+0000", 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1399999999999996e-07 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933522+0000", 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4010000000000001e-06 2024-09-17T16:02:43.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933536+0000", 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4269e-05 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.933544+0000", 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.576e-06 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938388+0000", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048441300000000003 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938478+0000", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9975000000000005e-05 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939194+0000", 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071552499999999995 2024-09-17T16:02:43.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944047+0000", 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048533990000000004 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320427+0000", 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.37638003799999997 2024-09-17T16:02:43.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:46.320482+0000", 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5390000000000003e-05 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:358 3.ds0 3:b27fc74f:::smithi18136759-24:head [copy-from ver 28] snapc 11=[11,10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e55)", 2024-09-17T16:02:43.093 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 49.652830246999997, 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.040742296999999997, 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 358 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.094 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376117+0000", 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376121+0000", 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1679999999999997e-06 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376135+0000", 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3054000000000001e-05 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.376141+0000", 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.443e-06 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381260+0000", 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005118671 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.381327+0000", 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7288999999999999e-05 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.392972+0000", 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011645071999999999 2024-09-17T16:02:43.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.396350+0000", 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033780709999999999 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416805+0000", 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020454785999999999 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:53.416859+0000", 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4045999999999997e-05 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.as0 63/32 ECSubWriteReply(tid=124, last_complete=63'41, committed=1, applied=1))", 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:43.100 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 41.601618158999997, 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45322852000000002, 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427329+0000", 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8100000000000002e-07 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.427336+0000", 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4069999999999998e-06 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880036+0000", 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.45270011199999999 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880042+0000", 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7950000000000004e-06 2024-09-17T16:02:43.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880109+0000", 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7659999999999999e-05 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:01.880557+0000", 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044786499999999998 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.104 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:509 3.ds0 3:bf204631:::smithi18136759-18:head [setxattr _header (57) in=7b] snapc 18=[18,17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.654105684000001, 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91880832899999998, 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.105 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 509 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374841+0000", 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374842+0000", 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9999999999999995e-07 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374847+0000", 2024-09-17T16:02:43.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.724e-06 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374855+0000", 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4400000000000005e-06 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374862+0000", 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999996e-06 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377155+0000", 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022931840000000002 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377217+0000", 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2512000000000001e-05 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.377509+0000", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000291057 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.381196+0000", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036871849999999999 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293578+0000", 2024-09-17T16:02:43.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91238186700000001 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293650+0000", 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1950000000000001e-05 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:510 3.ds0 3:bf204631:::smithi18136759-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e70)", 2024-09-17T16:02:43.111 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.654075798999997, 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.92014742599999999, 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 510 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374871+0000", 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374872+0000", 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5400000000000001e-07 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374875+0000", 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.083e-06 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374883+0000", 2024-09-17T16:02:43.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.374890+0000", 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9469999999999997e-06 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378503+0000", 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036131100000000001 2024-09-17T16:02:43.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378624+0000", 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121218 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294084+0000", 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91545977899999997 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.294190+0000", 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105797 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.295019+0000", 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082872900000000005 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 70/32 ECSubWriteReply(tid=171, last_complete=70'105, committed=1, applied=1))", 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 34.650372869000002, 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91541347900000003, 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:43.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378574+0000", 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378575+0000", 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6400000000000002e-07 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.378579+0000", 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0289999999999999e-06 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293447+0000", 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91486866300000003 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293456+0000", 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5040000000000002e-06 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293520+0000", 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3893999999999995e-05 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:09.293987+0000", 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000467725 2024-09-17T16:02:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 81/32 ECSubWriteReply(tid=246, last_complete=81'169, committed=1, applied=1))", 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 23.573810724000001, 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.121075089, 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455136+0000", 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455137+0000", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0999999999999997e-07 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.455142+0000", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9250000000000003e-06 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575575+0000", 2024-09-17T16:02:43.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.120433441 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575581+0000", 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5720000000000002e-06 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.575664+0000", 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3115000000000006e-05 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.576211+0000", 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054722600000000005 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 85/32 ECSubWriteReply(tid=275, last_complete=85'195, committed=1, applied=1))", 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 19.560277124999999, 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87648519400000002, 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468670+0000", 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3600000000000003e-07 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.468673+0000", 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0400000000000001e-06 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344179+0000", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87550525700000004 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344191+0000", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2237e-05 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.344247+0000", 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6391000000000003e-05 2024-09-17T16:02:43.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.345155+0000", 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090763300000000003 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:768 3.as0 3:5ccfabf6:::smithi18136759-44:head [setxattr _header (57) in=7b] snapc 20=[20,1f,1e,1d,1c,19,18,16,10,c,6] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.540593237, 2024-09-17T16:02:43.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.801372309, 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 768 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488354+0000", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5199999999999996e-07 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488358+0000", 2024-09-17T16:02:43.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4479999999999999e-06 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488366+0000", 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8380000000000002e-06 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488370+0000", 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2420000000000002e-06 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492136+0000", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037664959999999998 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492251+0000", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114357 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.492748+0000", 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00049682299999999997 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.494817+0000", 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002069286 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289552+0000", 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79473559500000002 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.289726+0000", 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017347200000000001 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:769 3.as0 3:5ccfabf6:::smithi18136759-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 18.540570747, 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80260884799999999, 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.136 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 769 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488376+0000", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488377+0000", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7099999999999997e-07 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488380+0000", 2024-09-17T16:02:43.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7069999999999998e-06 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488386+0000", 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4970000000000001e-06 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.488389+0000", 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9869999999999999e-06 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493011+0000", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046225939999999998 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:24.493109+0000", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8202999999999995e-05 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290108+0000", 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79699814300000005 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290237+0000", 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012928200000000001 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:25.290985+0000", 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000748064 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 98/32 ECSubWriteReply(tid=302, last_complete=98'219, committed=1, applied=1))", 2024-09-17T16:02:43.141 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:36.517735+0000", 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.511211823, 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.51274014000000001, 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.517735+0000", 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.517735+0000", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.517735+0000", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2e-07 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.517746+0000", 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0379e-05 2024-09-17T16:02:43.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:37.029783+0000", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.51203757100000002 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:37.029798+0000", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4273e-05 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:37.029909+0000", 2024-09-17T16:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111255 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:37.030475+0000", 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000566242 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:43.145 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:43.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:43.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 reset_purged_snaps_last 2024-09-17T16:02:43.148 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:43.145+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:43.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:43.247 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:43.246+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:43.268 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-17T16:02:43.268 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-17T16:02:43.268 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-17T16:02:43.268 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-17T16:02:43.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:43.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 reset_purged_snaps_last 2024-09-17T16:02:43.327 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:43.327 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:43.348 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:43.346+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:43.349 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:43.349 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:43.371 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:43.371 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:43.385 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:43.385 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:43.393 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:43.393 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:43.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:43.414 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:43.414 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:43.435 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:43.436 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:43.489 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:43.486+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:43.524 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:02:43.527 INFO:tasks.rados.rados.0.smithi181.stdout:357: done (5 left) 2024-09-17T16:02:43.527 INFO:tasks.rados.rados.0.smithi181.stdout:358: rollback oid 50 current snap is 40 2024-09-17T16:02:43.528 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 50 to 35 2024-09-17T16:02:43.528 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 2 to smithi18136759-36 2024-09-17T16:02:43.528 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 3 to smithi18136759-36 2024-09-17T16:02:43.528 INFO:tasks.rados.rados.0.smithi181.stdout:354: expect (ObjNum 108 snap 26 seq_num 108) 2024-09-17T16:02:43.543 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 2892420 2024-09-17T16:02:43.545 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:43.545 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:43.549 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:43.547+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:43.553 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1121310 2024-09-17T16:02:43.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:43.555 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:43.555 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:43.555 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:43.576 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:43.581 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:43.599 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:43.599 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 4 to smithi18136759-36 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 5 to smithi18136759-36 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:353: finishing rollback tid 0 to smithi18136759-4 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:353: finishing rollback tid 1 to smithi18136759-4 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:353: finishing rollback tid 2 to smithi18136759-4 2024-09-17T16:02:43.621 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 102 (ObjNum 83 snap 20 seq_num 83) dirty exists 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:355: finishing copy_from racing read to smithi18136759-31 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: finishing write tid 6 to smithi18136759-36 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: oid 36 updating version 0 to 142 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: oid 36 version 142 is already newer than 141 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: oid 36 version 142 is already newer than 139 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: oid 36 updating version 142 to 143 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: oid 36 version 143 is already newer than 140 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 36 v 143 (ObjNum 133 snap 40 seq_num 133) dirty exists 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:352: left oid 36 (ObjNum 133 snap 40 seq_num 133) 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 1 to smithi18136759-44 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 2 to smithi18136759-44 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:355: finishing copy_from to smithi18136759-31 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 2 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T16:02:43.622 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 3 to smithi18136759-44 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:352: done (5 left) 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:353: done (4 left) 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:354: done (3 left) 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:355: done (2 left) 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:359: delete oid 48 current snap is 40 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 4 to smithi18136759-44 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 5 to smithi18136759-44 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: finishing write tid 6 to smithi18136759-44 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 44 updating version 0 to 58 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 44 updating version 58 to 60 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 44 updating version 60 to 62 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 44 version 62 is already newer than 61 2024-09-17T16:02:43.623 INFO:tasks.rados.rados.0.smithi181.stdout:356: oid 44 version 62 is already newer than 59 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 62 (ObjNum 134 snap 40 seq_num 134) dirty exists 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:356: left oid 44 (ObjNum 134 snap 40 seq_num 134) 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:358: finishing rollback tid 0 to smithi18136759-50 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:358: finishing rollback tid 1 to smithi18136759-50 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:358: finishing rollback tid 2 to smithi18136759-50 2024-09-17T16:02:43.624 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 147 (ObjNum 128 snap 34 seq_num 128) dirty exists 2024-09-17T16:02:43.624 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:43.624 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:43.628 INFO:tasks.rados.rados.0.smithi181.stdout:356: done (2 left) 2024-09-17T16:02:43.628 INFO:tasks.rados.rados.0.smithi181.stdout:358: done (1 left) 2024-09-17T16:02:43.628 INFO:tasks.rados.rados.0.smithi181.stdout:359: done (0 left) 2024-09-17T16:02:43.628 INFO:tasks.rados.rados.0.smithi181.stdout:360: snap_remove snap 26 2024-09-17T16:02:43.650 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:43.649+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:43.705 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:43.705 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:43.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:43.752 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:43.750+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:43.846 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:43.846 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:43.852 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:43.850+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:43.861 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:43.861 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:43.867 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:43.867 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:43.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:43.888 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:43.888 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:43.909 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:43.907+0000 7f3f3bfff640 0 -- 172.21.15.6:0/2533451210 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f3f3c221810 msgr2=0x7f3f3c223c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-17T16:02:43.953 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:43.951+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:44.017 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.017 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:44.055 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:44.053+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:44.132 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.132 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.133 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:44.133 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:44.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:44.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:44.180 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:44.176+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:44.247 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.248 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:44.257 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:44.254+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:44.358 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:44.355+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.365 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:44.458 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:44.456+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:44.481 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.481 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:44.537 INFO:tasks.rados.rados.0.smithi181.stdout:360: done (0 left) 2024-09-17T16:02:44.537 INFO:tasks.rados.rados.0.smithi181.stdout:361: copy_from oid 4 from oid 33 current snap is 40 2024-09-17T16:02:44.537 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 19 current snap is 40 2024-09-17T16:02:44.537 INFO:tasks.rados.rados.0.smithi181.stdout:362: seq_num 136 ranges {770048=737280,1507328=770048,2277376=540672,2818048=245760} 2024-09-17T16:02:44.546 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18136759-19 from 770048 to 1507328 tid 1 2024-09-17T16:02:44.552 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18136759-19 from 1507328 to 2277376 tid 2 2024-09-17T16:02:44.555 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18136759-19 from 2277376 to 2818048 tid 3 2024-09-17T16:02:44.557 INFO:tasks.rados.rados.0.smithi181.stdout:362: writing smithi18136759-19 from 2818048 to 3063808 tid 4 2024-09-17T16:02:44.557 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 31 current snap is 40 2024-09-17T16:02:44.557 INFO:tasks.rados.rados.0.smithi181.stdout:363: seq_num 137 ranges {311296=753664,1064960=786432,1851392=655360,2506752=114688} 2024-09-17T16:02:44.558 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:44.557+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:44.563 INFO:tasks.rados.rados.0.smithi181.stdout:363: writing smithi18136759-31 from 311296 to 1064960 tid 1 2024-09-17T16:02:44.568 INFO:tasks.rados.rados.0.smithi181.stdout:363: writing smithi18136759-31 from 1064960 to 1851392 tid 2 2024-09-17T16:02:44.571 INFO:tasks.rados.rados.0.smithi181.stdout:363: writing smithi18136759-31 from 1851392 to 2506752 tid 3 2024-09-17T16:02:44.572 INFO:tasks.rados.rados.0.smithi181.stdout:363: writing smithi18136759-31 from 2506752 to 2621440 tid 4 2024-09-17T16:02:44.572 INFO:tasks.rados.rados.0.smithi181.stdout:364: snap_remove snap 22 2024-09-17T16:02:44.612 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.612 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:44.660 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:44.658+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:44.761 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:44.759+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:44.769 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.769 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:44.861 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:44.860+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:44.891 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:44.891 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:44.900 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:44.963 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:44.961+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:45.003 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.003 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.012 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:45.064 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:45.061+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:45.156 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.156 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.165 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:45.165 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:45.163+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:45.266 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:45.263+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:45.269 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.269 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:45.367 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:45.365+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:45.382 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.382 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.392 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:45.468 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:45.466+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:45.535 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.535 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:02:45.545 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:364: done (3 left) 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:365: rollback oid 44 current snap is 40 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 44 to 10 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 1 to smithi18136759-19 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 2 to smithi18136759-19 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 3 to smithi18136759-19 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 4 to smithi18136759-19 2024-09-17T16:02:45.549 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 5 to smithi18136759-19 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: finishing write tid 6 to smithi18136759-19 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 19 updating version 0 to 159 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 19 updating version 159 to 161 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 19 version 161 is already newer than 160 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 19 version 161 is already newer than 155 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: oid 19 version 161 is already newer than 156 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 161 (ObjNum 136 snap 40 seq_num 136) dirty exists 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: left oid 19 (ObjNum 136 snap 40 seq_num 136) 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 1 to smithi18136759-31 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 2 to smithi18136759-31 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 3 to smithi18136759-31 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:362: done (3 left) 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:366: snap_create 2024-09-17T16:02:45.550 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 4 to smithi18136759-31 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:361: finishing copy_from racing read to smithi18136759-4 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 5 to smithi18136759-31 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:361: finishing copy_from to smithi18136759-4 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 4 v 44 (ObjNum 120 snap 29 seq_num 120) dirty exists 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: finishing write tid 6 to smithi18136759-31 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: oid 31 updating version 0 to 46 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: oid 31 updating version 46 to 47 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: oid 31 updating version 47 to 49 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: oid 31 version 49 is already newer than 45 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: oid 31 version 49 is already newer than 48 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 31 v 49 (ObjNum 137 snap 40 seq_num 137) dirty exists 2024-09-17T16:02:45.551 INFO:tasks.rados.rados.0.smithi181.stdout:363: left oid 31 (ObjNum 137 snap 40 seq_num 137) 2024-09-17T16:02:45.563 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing rollback tid 0 to smithi18136759-44 2024-09-17T16:02:45.563 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing rollback tid 1 to smithi18136759-44 2024-09-17T16:02:45.564 INFO:tasks.rados.rados.0.smithi181.stdout:365: finishing rollback tid 2 to smithi18136759-44 2024-09-17T16:02:45.564 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 65 (ObjNum 59 snap 7 seq_num 59) dirty exists 2024-09-17T16:02:45.588 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:45.586+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:45.653 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.653 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:45.669 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:45.667+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:45.769 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:45.767+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:45.803 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.804 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.813 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:45.871 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:45.869+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:45.916 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:45.916 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:45.924 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:45.971 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:45.970+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:46.027 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:46.027 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:46.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:46.072 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:46.070+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:46.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:46.178 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:46.178 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:46.182 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:46.179+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:46.187 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:46.274 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:46.271+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:46.276 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:46.276 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:46.291 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:46.291 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:46.299 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:46.299 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:46.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:46.321 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:46.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:46.342 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:46.343 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:46.362 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:46.375 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:46.373+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:46.384 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:46.385 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:46.405 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:46.406 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:46.406 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:46.416 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:46.416 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:46.416 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:46.416 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:46.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:46.426 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:46.426 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:46.448 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:46.448 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:46.469 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:46.469 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:46.475 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:46.473+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:46.490 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:46.490 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:46.510 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:46.511 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:46.529 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:46.532 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:46.533 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:46.533 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:46.530+0000 7ff5aaffd640 0 -- 172.21.15.6:0/3468886438 >> [v2:172.21.15.169:6800/4170280864,v1:172.21.15.169:6801/4170280864] conn(0x7ff5ac21cd30 msgr2=0x7ff5ac21f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:46.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:46.541 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:361: done (3 left) 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:363: done (2 left) 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:365: done (1 left) 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:366: done (0 left) 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:367: read oid 39 snap -1 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:367: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-17T16:02:46.545 INFO:tasks.rados.rados.0.smithi181.stdout:368: snap_create 2024-09-17T16:02:46.552 INFO:tasks.rados.rados.0.smithi181.stdout:367: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-17T16:02:46.555 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 522112 2024-09-17T16:02:46.556 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:46.563 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 183486 2024-09-17T16:02:46.576 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:46.574+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:46.642 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:46.642 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=167, reqid=client.4560.0:228, at_version=44'14, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 63.731261603999997, 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045964070000000003, 2024-09-17T16:02:46.643 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:46.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908482+0000", 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908499+0000", 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6748999999999998e-05 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908533+0000", 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3831999999999997e-05 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908539+0000", 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5980000000000002e-06 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908619+0000", 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740999999999994e-05 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908629+0000", 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913043+0000", 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044149380000000002 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913078+0000", 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4094999999999998e-05 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.711998422999997, 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046997530000000001, 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.649 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927762+0000", 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7229000000000001e-05 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927945+0000", 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018362200000000001 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928161+0000", 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021584100000000001 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928170+0000", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6249999999999996e-06 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928242+0000", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2434999999999993e-05 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928253+0000", 2024-09-17T16:02:46.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932409+0000", 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041564169999999999 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932444+0000", 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.711793397000001, 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048629329999999998, 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927950+0000", 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3399999999999999e-07 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928041+0000", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0953999999999997e-05 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928205+0000", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000164537 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928213+0000", 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.272e-06 2024-09-17T16:02:46.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929483+0000", 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001270106 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929493+0000", 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0207000000000001e-05 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932778+0000", 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032847419999999998 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932812+0000", 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4581e-05 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=12, reqid=client.4560.0:273, at_version=45'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:46.658 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.711661878000001, 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043524990000000001, 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928082+0000", 2024-09-17T16:02:46.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2799999999999996e-07 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928098+0000", 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6070999999999999e-05 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928229+0000", 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013130999999999999 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928234+0000", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928583+0000", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034869300000000002 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928595+0000", 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1433e-05 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932398+0000", 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038034890000000002 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932433+0000", 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5386999999999998e-05 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.663 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 61.720145193999997, 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043836329999999996, 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919621+0000", 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2204e-05 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919652+0000", 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1846e-05 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919661+0000", 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3075000000000002e-05 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919733+0000", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3540000000000006e-06 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923937+0000", 2024-09-17T16:02:46.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042032980000000003 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923981+0000", 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4685000000000003e-05 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=79, reqid=client.4560.0:332, at_version=52'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:46.668 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 56.282128366000002, 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051054819999999997, 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357615+0000", 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-07 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358043+0000", 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042830599999999999 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358075+0000", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2512999999999999e-05 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358085+0000", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0081000000000001e-05 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358170+0000", 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4362000000000003e-05 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358181+0000", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1535000000000001e-05 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362680+0000", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989069999999999 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362720+0000", 2024-09-17T16:02:46.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9641000000000002e-05 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=197, reqid=client.4560.0:392, at_version=60'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.289370857999998, 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0099856809999999997, 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:46.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2599999999999999e-07 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350537+0000", 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016478400000000001 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350604+0000", 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7305999999999996e-05 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350615+0000", 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1154e-05 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350701+0000", 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6136000000000003e-05 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350721+0000", 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9967e-05 2024-09-17T16:02:46.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360314+0000", 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0095931230000000003 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360358+0000", 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3084999999999998e-05 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=198, reqid=client.4560.0:393, at_version=60'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.286845141999997, 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075206600000000002, 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352899+0000", 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1230000000000001e-06 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353041+0000", 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014228299999999999 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353091+0000", 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9942999999999999e-05 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353101+0000", 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353167+0000", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5365999999999997e-05 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353178+0000", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1429e-05 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360364+0000", 2024-09-17T16:02:46.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071857500000000003 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360418+0000", 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4317000000000002e-05 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=199, reqid=client.4560.0:394, at_version=60'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.285156591000003, 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058628329999999996, 2024-09-17T16:02:46.682 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354587+0000", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4099999999999997e-07 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354652+0000", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5085000000000005e-05 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354702+0000", 2024-09-17T16:02:46.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9641999999999997e-05 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354711+0000", 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0869999999999996e-06 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354774+0000", 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3048999999999999e-05 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354784+0000", 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163000000000001e-05 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360427+0000", 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056429729999999999 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360449+0000", 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1993000000000001e-05 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.686 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=200, reqid=client.4560.0:395, at_version=60'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.283730452, 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989890000000001, 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356013+0000", 2024-09-17T16:02:46.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356031+0000", 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7592999999999999e-05 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356068+0000", 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7345999999999998e-05 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356079+0000", 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0730999999999999e-05 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356150+0000", 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1668999999999994e-05 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356161+0000", 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1027e-05 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360478+0000", 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043165800000000004 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360511+0000", 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3251999999999999e-05 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.301486738999998, 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046971620000000004, 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338276+0000", 2024-09-17T16:02:46.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9932000000000002e-05 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338298+0000", 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1336e-05 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338306+0000", 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6810000000000004e-06 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338467+0000", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160538 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338477+0000", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5929999999999995e-06 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342913+0000", 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044366149999999997 2024-09-17T16:02:46.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342953+0000", 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0061000000000002e-05 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=134, reqid=osd.13.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=64'15))", 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.289695979000001, 2024-09-17T16:02:46.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042510439999999998, 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:46.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350056+0000", 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350080+0000", 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4360999999999999e-05 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350089+0000", 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2609999999999993e-06 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350189+0000", 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100868 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350199+0000", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1679999999999993e-06 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354267+0000", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040685629999999999 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354298+0000", 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0626000000000001e-05 2024-09-17T16:02:46.700 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 64/53 ECSubWrite(tid=211, reqid=osd.11.0:211, at_version=64'37, trim_to=0'0, roll_forward_to=62'35))", 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 44.280676606999997, 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048593489999999998, 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.701 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359067+0000", 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6699999999999999e-07 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359087+0000", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0594999999999998e-05 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359129+0000", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1458999999999997e-05 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359138+0000", 2024-09-17T16:02:46.703 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6050000000000004e-06 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359207+0000", 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8885000000000002e-05 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359218+0000", 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0524e-05 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.704 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363901+0000", 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046832100000000002 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363925+0000", 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4703999999999999e-05 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:46.705 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.28871796, 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059954810000000004, 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.706 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351050+0000", 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4579e-05 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.707 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351200+0000", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015020800000000001 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351208+0000", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6799999999999999e-06 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352434+0000", 2024-09-17T16:02:46.708 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012252350000000001 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352445+0000", 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1440000000000001e-05 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.356973+0000", 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045277900000000003 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.709 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.357020+0000", 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=102, reqid=client.4560.0:502, at_version=69'50, trim_to=0'0, roll_forward_to=69'46))", 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.229736971999998, 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042988599999999998, 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.710 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.711 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.37e-07 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410021+0000", 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4741e-05 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410042+0000", 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0798e-05 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.712 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410047+0000", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5459999999999999e-06 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410084+0000", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6556e-05 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410089+0000", 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4600000000000002e-06 2024-09-17T16:02:46.713 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414267+0000", 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041775720000000001 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414305+0000", 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.765e-05 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.714 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 81/53 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=81'76, trim_to=0'0, roll_forward_to=77'72))", 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.200219270000002, 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048353479999999997, 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.715 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439524+0000", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439546+0000", 2024-09-17T16:02:46.716 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1430999999999999e-05 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439578+0000", 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962000000000003e-05 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439590+0000", 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2201999999999999e-05 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.717 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439662+0000", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2361999999999996e-05 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439675+0000", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2848e-05 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444304+0000", 2024-09-17T16:02:46.718 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046291250000000004 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444359+0000", 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4648000000000003e-05 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.14s3 81/53 ECSubWrite(tid=39, reqid=osd.1.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=81'30))", 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:46.719 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 27.199978816000002, 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046130629999999997, 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:46.720 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439765+0000", 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.001e-06 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439778+0000", 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3295999999999999e-05 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.721 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439815+0000", 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6511000000000003e-05 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439828+0000", 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.329e-05 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.722 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439899+0000", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1125999999999996e-05 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439911+0000", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1792e-05 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444330+0000", 2024-09-17T16:02:46.723 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044186609999999999 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444377+0000", 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=241, reqid=client.4560.0:860, at_version=104'100, trim_to=0'0, roll_forward_to=102'98))", 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:46.724 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0989514160000002, 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051699650000000003, 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.725 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540792+0000", 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540823+0000", 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1467999999999998e-05 2024-09-17T16:02:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540874+0000", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0460999999999997e-05 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540884+0000", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540957+0000", 2024-09-17T16:02:46.727 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3083000000000005e-05 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540969+0000", 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1712000000000001e-05 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545929+0000", 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049599020000000004 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.728 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545961+0000", 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2221999999999997e-05 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=242, reqid=client.4560.0:861, at_version=104'101, trim_to=0'0, roll_forward_to=102'98))", 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0985876819999998, 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048239809999999998, 2024-09-17T16:02:46.729 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.730 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541156+0000", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7400000000000003e-07 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541183+0000", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7566999999999999e-05 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541215+0000", 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2101000000000003e-05 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.731 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541223+0000", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541924+0000", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070091800000000003 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541935+0000", 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0835999999999999e-05 2024-09-17T16:02:46.732 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545966+0000", 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040307620000000002 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545979+0000", 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3123e-05 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.733 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=245, reqid=client.4560.0:862, at_version=0'0, trim_to=0'0, roll_forward_to=104'102))", 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 4.0922773460000004, 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.63482562300000001, 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.734 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547466+0000", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547476+0000", 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6709999999999994e-06 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.735 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181369+0000", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.63389281099999994 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181396+0000", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.741e-05 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181516+0000", 2024-09-17T16:02:46.736 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120091 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181539+0000", 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2106000000000001e-05 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.182254+0000", 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071494500000000003 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:46.737 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.182291+0000", 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7512999999999998e-05 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:46.738 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:46.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:46.740 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:46.737+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:46.755 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:46.755 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:46.776 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:46.776 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:46.780 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:46.840 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:46.838+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:46.883 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:46.883 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:46.883 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:46.883 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:46.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:46.940 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:46.938+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:46.995 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:46.995 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:46.996 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:46.996 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:46.996 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:47.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:47.041 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:47.039+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:47.109 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 90.357144144000003, 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19491629299999999, 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748958+0000", 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748966+0000", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5870000000000004e-06 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748979+0000", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748988+0000", 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749052+0000", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4618999999999999e-05 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749064+0000", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.939947+0000", 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19088328399999999 2024-09-17T16:02:47.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940079+0000", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013157500000000001 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940245+0000", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016577900000000001 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943345+0000", 2024-09-17T16:02:47.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031004499999999998 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943806+0000", 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046112200000000001 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943843+0000", 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6578000000000001e-05 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943873+0000", 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [cmpxattr striper.excl (25) op 1 mode 1 in=12b,setxattr striper.excl in=12b,call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 90.158058818000001, 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83632086699999997, 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.116 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948043+0000", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948048+0000", 2024-09-17T16:02:47.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948060+0000", 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1761e-05 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948066+0000", 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5350000000000001e-06 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948111+0000", 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4783000000000001e-05 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948190+0000", 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948399+0000", 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000209292 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948922+0000", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052251800000000003 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784264+0000", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83534246599999995 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784309+0000", 2024-09-17T16:02:47.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4660000000000003e-05 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784363+0000", 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4095000000000003e-05 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.4219.0:8 1.0 e24/23 ondisk, result = 0)", 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 90.156936623000007, 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83521008600000002, 2024-09-17T16:02:47.122 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949165+0000", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949169+0000", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9339999999999999e-06 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784135+0000", 2024-09-17T16:02:47.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83496651200000005 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784151+0000", 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784250+0000", 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8222999999999998e-05 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784260+0000", 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0265000000000001e-05 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784374+0000", 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114373 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4296.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-17T16:02:47.126 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 88.346748410000004, 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.241196405, 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4296", 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.6:0/1428345084", 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759356+0000", 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.484e-06 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759366+0000", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9089999999999995e-06 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759375+0000", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759397+0000", 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1824e-05 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.129 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759444+0000", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6539000000000003e-05 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759449+0000", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998167+0000", 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23871730099999999 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998242+0000", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4892000000000001e-05 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998303+0000", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0931999999999999e-05 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998708+0000", 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040511399999999998 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000499+0000", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017911909999999999 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000526+0000", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6798999999999999e-05 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000549+0000", 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3110000000000001e-05 2024-09-17T16:02:47.132 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:34 3.2s0 3:41aa4422:::smithi18136759-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.249485906000004, 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015312129000000001, 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.133 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 34 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856616+0000", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4899999999999999e-06 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856732+0000", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011513 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856745+0000", 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3677e-05 2024-09-17T16:02:47.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856753+0000", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856824+0000", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1291999999999994e-05 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856962+0000", 2024-09-17T16:02:47.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013730799999999999 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.857708+0000", 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074624700000000001 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861225+0000", 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035171249999999999 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871848+0000", 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010623323 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871927+0000", 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8844000000000006e-05 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:35 3.2s0 3:41aa4422:::smithi18136759-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:47.138 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.249330365999995, 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015326866, 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 35 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:47.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856771+0000", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856775+0000", 2024-09-17T16:02:47.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.33e-06 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856782+0000", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0800000000000002e-06 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856786+0000", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5759999999999999e-06 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858308+0000", 2024-09-17T16:02:47.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015218009999999999 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858393+0000", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5502999999999998e-05 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859025+0000", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000632009 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861267+0000", 2024-09-17T16:02:47.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022417639999999998 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872026+0000", 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010758772999999999 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872097+0000", 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1390000000000006e-05 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:36 3.2s0 3:41aa4422:::smithi18136759-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.249245023, 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017199677, 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 36 2024-09-17T16:02:47.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.145 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856861+0000", 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0759999999999996e-06 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856869+0000", 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6319999999999997e-06 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856876+0000", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5989999999999998e-06 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859529+0000", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026531060000000001 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859628+0000", 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9104000000000005e-05 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872415+0000", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012787056 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872500+0000", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999998e-05 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874055+0000", 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015554810000000001 2024-09-17T16:02:47.148 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=11, reqid=client.4560.0:50, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.235161357999999, 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016547899000000001, 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870940+0000", 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870953+0000", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2798e-05 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881713+0000", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010760202 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881721+0000", 2024-09-17T16:02:47.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8949999999999998e-06 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881765+0000", 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3778e-05 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881775+0000", 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0135999999999999e-05 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887451+0000", 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056764090000000003 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887487+0000", 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6065000000000001e-05 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.153 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:81 3.2s0 3:4c2fa52d:::smithi18136759-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.212057559000002, 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015999928, 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 81 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.154 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894044+0000", 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:47.155 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894045+0000", 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065e-06 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894048+0000", 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.489e-06 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.156 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894051+0000", 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9799999999999998e-06 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896400+0000", 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023486919999999999 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896473+0000", 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3380000000000006e-05 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908531+0000", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012058016 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908615+0000", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3968000000000007e-05 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.910043+0000", 2024-09-17T16:02:47.158 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014280569999999999 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s3 35/32 ECSubWrite(tid=34, reqid=client.4560.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.180701708000001, 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021043124999999999, 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.159 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925400+0000", 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:47.160 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925410+0000", 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0345e-05 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944975+0000", 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019565013999999999 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.161 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944985+0000", 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0257e-05 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945787+0000", 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00080139199999999995 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945797+0000", 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9590000000000007e-06 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.162 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946410+0000", 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061359000000000001 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946442+0000", 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=42, reqid=client.4560.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:47.163 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.150343684000006, 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022376667999999999, 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.164 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955758+0000", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955765+0000", 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.345e-06 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.165 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977438+0000", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021672416999999999 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977447+0000", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977511+0000", 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4066000000000006e-05 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.166 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977519+0000", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8329999999999994e-06 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978105+0000", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058612099999999999 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978134+0000", 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8558e-05 2024-09-17T16:02:47.167 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:268 3.2s0 3:406bd9c9:::smithi18136759-23:head [append 0~720896] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 63.182777180000002, 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016657371000000001, 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.168 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 268 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923326+0000", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0999999999999998e-06 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923806+0000", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048034 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923821+0000", 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5180999999999999e-05 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923831+0000", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4450000000000002e-06 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923891+0000", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9858999999999997e-05 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923971+0000", 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0493000000000003e-05 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.924982+0000", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001010543 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929374+0000", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043921699999999999 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939921+0000", 2024-09-17T16:02:47.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010546754 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939981+0000", 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0486000000000003e-05 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:271 3.2s0 3:406bd9c9:::smithi18136759-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 63.180141110999998, 2024-09-17T16:02:47.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015374301, 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 271 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925961+0000", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925964+0000", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5319999999999998e-06 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925976+0000", 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1795e-05 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925983+0000", 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.289e-06 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929232+0000", 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032491070000000002 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929352+0000", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119639 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940555+0000", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011203489000000001 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940634+0000", 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8165999999999997e-05 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.941334+0000", 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000700496 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:392 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~589824] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.763624481000001, 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018219264999999998, 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 392 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:47.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342479+0000", 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.481e-06 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343310+0000", 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083096699999999995 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343328+0000", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8059000000000001e-05 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343340+0000", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1664e-05 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343445+0000", 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010582900000000001 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343453+0000", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348803+0000", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053503420000000001 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348975+0000", 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017183899999999999 2024-09-17T16:02:47.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350509+0000", 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015333320000000001 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356400+0000", 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058917320000000002 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360612+0000", 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042120680000000002 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360696+0000", 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:396 3.2s0 3:465e1f11:::smithi18136759-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:47.184 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 48.758363074999998, 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015698437999999999, 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 396 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8699999999999995e-07 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347741+0000", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6690000000000002e-06 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347747+0000", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.455e-06 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347752+0000", 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0459999999999996e-06 2024-09-17T16:02:47.187 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356238+0000", 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084855650000000005 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356325+0000", 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7039999999999993e-05 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.188 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361673+0000", 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053483849999999998 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361762+0000", 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8825000000000004e-05 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.363436+0000", 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001673966 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:484 3.18s0 3:19aefe3f:::smithi18136759-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.755142974999998, 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017624631000000002, 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 484 2024-09-17T16:02:47.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350961+0000", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352262+0000", 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001300708 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352278+0000", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5848000000000001e-05 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354256+0000", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019783330000000001 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354327+0000", 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0839000000000004e-05 2024-09-17T16:02:47.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363568+0000", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092410559999999992 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363668+0000", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9382000000000005e-05 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368582+0000", 2024-09-17T16:02:47.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049148810000000003 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:488 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [setxattr _header (57) in=64b] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.741981039999999, 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83388352600000004, 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.195 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 488 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364123+0000", 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9220000000000001e-06 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364131+0000", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364137+0000", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4679999999999997e-06 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366664+0000", 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025266949999999998 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366744+0000", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0739000000000001e-05 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367168+0000", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042360999999999999 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368066+0000", 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089781599999999996 2024-09-17T16:02:47.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197852+0000", 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82978574900000002 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198003+0000", 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015172699999999999 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:489 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.741954819999997, 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83494334199999998, 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 489 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.201 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364147+0000", 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2024-09-17T16:02:47.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364149+0000", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2359999999999999e-06 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364156+0000", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4630000000000004e-06 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364163+0000", 2024-09-17T16:02:47.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368609+0000", 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044454869999999997 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368658+0000", 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9438999999999999e-05 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198316+0000", 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965781999999999 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198436+0000", 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012003299999999999 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.199089+0000", 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000653445 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.18s0 69/53 ECSubWriteReply(tid=224, last_complete=69'42, committed=1, applied=1))", 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 39.738176103000001, 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83034511799999999, 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.13e-07 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367929+0000", 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197585+0000", 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965586700000005 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197599+0000", 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4369000000000001e-05 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197711+0000", 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011220199999999999 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198270+0000", 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00055871000000000002 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.19s1 86/53 ECSubRead(tid=180, to_read={3:9b2471fe:::smithi18136759-32:17=114688,16384,0}, subchunks={3:9b2471fe:::smithi18136759-32:17=[0,1]}, attrs_to_read=))", 2024-09-17T16:02:47.209 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 22.643075563, 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43055667600000003, 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463026+0000", 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.09e-07 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463035+0000", 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893268+0000", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.430232958 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893300+0000", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2008999999999998e-05 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893479+0000", 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017834700000000001 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893582+0000", 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010328900000000001 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:47.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:47.215 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:47.212+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:47.316 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:47.314+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:47.317 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:47.317 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:47.317 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:47.317 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:47.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:47.416 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:47.415+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:47.428 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:47.428 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:47.428 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:47.428 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:47.428 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:47.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:47.517 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:47.515+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:47.547 INFO:tasks.rados.rados.0.smithi181.stdout:367: done (1 left) 2024-09-17T16:02:47.547 INFO:tasks.rados.rados.0.smithi181.stdout:368: done (0 left) 2024-09-17T16:02:47.547 INFO:tasks.rados.rados.0.smithi181.stdout:369: delete oid 29 current snap is 42 2024-09-17T16:02:47.581 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:47.581 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.711449896000005, 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.582 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.583 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:02:47.584 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.585 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:47.586 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.711060184000004, 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.587 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.588 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:02:47.589 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.590 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.701639775000004, 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.591 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:02:47.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.595 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646954249999993, 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.596 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:02:47.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:02:47.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646571851000004, 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:02:47.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:02:47.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646536057999995, 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.607 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:02:47.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:02:47.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.612 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646507893000006, 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:02:47.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646234374000002, 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.618 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:02:47.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:02:47.623 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.646223317999997, 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.624 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:02:47.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.613468165, 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:02:47.629 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:02:47.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:02:47.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:02:47.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.612979115000002, 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.635 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.638 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:02:47.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 73.612938264999997, 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.641 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:02:47.644 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.665218668999998, 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073441670000000004, 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.646 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913695+0000", 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913725+0000", 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9743999999999998e-05 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919758+0000", 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00603318 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919766+0000", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3610000000000001e-06 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919835+0000", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9336000000000002e-05 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919844+0000", 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:02:47.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921001+0000", 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115653 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921039+0000", 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7700000000000002e-05 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.650 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 46/32 ECSubWriteReply(tid=17, last_complete=46'12, committed=1, applied=1))", 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.655232091999999, 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.330502611, 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:47.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923682+0000", 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923686+0000", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253962+0000", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33027621400000001 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253975+0000", 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-17T16:02:47.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254119+0000", 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014402100000000001 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254184+0000", 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4158999999999993e-05 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:346 3.3s0 3:c1daf1ed:::smithi18136759-38:head [delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 55.220143270999998, 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069854690000000002, 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.655 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358772+0000", 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0810000000000001e-06 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358779+0000", 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2849999999999997e-06 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358790+0000", 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9759999999999995e-06 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358838+0000", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9730000000000001e-05 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358966+0000", 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012749199999999999 2024-09-17T16:02:47.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.359434+0000", 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046890000000000001 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.362289+0000", 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028544030000000001 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:47.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365699+0000", 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034100210000000001 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365755+0000", 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6456000000000003e-05 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.10s3 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:47.660 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.232771831000001, 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081444710000000003, 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-07 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346159+0000", 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7445e-05 2024-09-17T16:02:47.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346307+0000", 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014812199999999999 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346314+0000", 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.477e-06 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346436+0000", 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012286999999999999 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346445+0000", 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.833e-06 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354242+0000", 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077967399999999999 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.664 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354286+0000", 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3801999999999998e-05 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.16s2 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.230440010000002, 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059176979999999999, 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.665 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348474+0000", 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5799999999999999e-07 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348485+0000", 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0892e-05 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348510+0000", 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5287999999999999e-05 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348517+0000", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.267e-06 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348561+0000", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4360000000000002e-05 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348568+0000", 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7270000000000001e-06 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354355+0000", 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057869480000000001 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354391+0000", 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5658000000000003e-05 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.14s4 81/53 ECSubWrite(tid=38, reqid=osd.1.0:38, at_version=81'30, trim_to=0'0, roll_forward_to=79'28))", 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.151328002, 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071372670000000001, 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.48e-07 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427607+0000", 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2113e-05 2024-09-17T16:02:47.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427687+0000", 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9518000000000007e-05 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427696+0000", 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6680000000000007e-06 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427760+0000", 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4349000000000003e-05 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427770+0000", 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0236e-05 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.434677+0000", 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069065589999999996 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.434722+0000", 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5575999999999999e-05 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.19s3 81/53 ECSubWrite(tid=148, reqid=osd.14.0:148, at_version=81'37, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.438866+0000", 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.140046943000002, 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058712130000000001, 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.674 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.438866+0000", 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.438866+0000", 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.438866+0000", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3200000000000001e-07 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.438889+0000", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2779999999999999e-05 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.441095+0000", 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022056340000000002 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.441104+0000", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9840000000000007e-06 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.441174+0000", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9762000000000001e-05 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.441185+0000", 2024-09-17T16:02:47.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1078e-05 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444689+0000", 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035046220000000002 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444737+0000", 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8121000000000002e-05 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.19s3 81/53 ECSubWrite(tid=157, reqid=client.4560.0:672, at_version=81'43, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 28.126433447, 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061579099999999999, 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452481+0000", 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:47.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452813+0000", 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033278099999999998 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452847+0000", 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3637000000000003e-05 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452853+0000", 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7309999999999998e-06 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452933+0000", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0384999999999997e-05 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452944+0000", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0900000000000001e-05 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.458602+0000", 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005657825 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:47.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.458638+0000", 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781000000000002e-05 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:47.683 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:47.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:47.685 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:47.682+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:47.785 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:47.782+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:47.806 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:47.806 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:47.806 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:47.806 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:47.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:47.879 INFO:tasks.rados.rados.0.smithi181.stdout:369: done (0 left) 2024-09-17T16:02:47.879 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 2 current snap is 42 2024-09-17T16:02:47.880 INFO:tasks.rados.rados.0.smithi181.stdout:370: seq_num 139 ranges {1966080=786432,2752512=688128,3440640=475136,3915776=393216} 2024-09-17T16:02:47.886 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:47.884+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:47.894 INFO:tasks.rados.rados.0.smithi181.stdout:370: writing smithi18136759-2 from 1966080 to 2752512 tid 1 2024-09-17T16:02:47.897 INFO:tasks.rados.rados.0.smithi181.stdout:370: writing smithi18136759-2 from 2752512 to 3440640 tid 2 2024-09-17T16:02:47.900 INFO:tasks.rados.rados.0.smithi181.stdout:370: writing smithi18136759-2 from 3440640 to 3915776 tid 3 2024-09-17T16:02:47.903 INFO:tasks.rados.rados.0.smithi181.stdout:370: writing smithi18136759-2 from 3915776 to 4308992 tid 4 2024-09-17T16:02:47.903 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 50 current snap is 42 2024-09-17T16:02:47.903 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 1 to smithi18136759-2 2024-09-17T16:02:47.903 INFO:tasks.rados.rados.0.smithi181.stdout:371: seq_num 140 ranges {4079616=802816,4882432=425984,5308416=720896,6029312=409600} 2024-09-17T16:02:47.929 INFO:tasks.rados.rados.0.smithi181.stdout:371: writing smithi18136759-50 from 4079616 to 4882432 tid 1 2024-09-17T16:02:47.931 INFO:tasks.rados.rados.0.smithi181.stdout:371: writing smithi18136759-50 from 4882432 to 5308416 tid 2 2024-09-17T16:02:47.935 INFO:tasks.rados.rados.0.smithi181.stdout:371: writing smithi18136759-50 from 5308416 to 6029312 tid 3 2024-09-17T16:02:47.938 INFO:tasks.rados.rados.0.smithi181.stdout:371: writing smithi18136759-50 from 6029312 to 6438912 tid 4 2024-09-17T16:02:47.938 INFO:tasks.rados.rados.0.smithi181.stdout:372: snap_create 2024-09-17T16:02:47.938 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 2 to smithi18136759-2 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 3 to smithi18136759-2 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 4 to smithi18136759-2 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 5 to smithi18136759-2 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: finishing write tid 6 to smithi18136759-2 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: oid 2 updating version 0 to 27 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: oid 2 updating version 27 to 28 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: oid 2 version 28 is already newer than 26 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: oid 2 updating version 28 to 29 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: oid 2 updating version 29 to 30 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 2 v 30 (ObjNum 139 snap 42 seq_num 139) dirty exists 2024-09-17T16:02:47.939 INFO:tasks.rados.rados.0.smithi181.stdout:370: left oid 2 (ObjNum 139 snap 42 seq_num 139) 2024-09-17T16:02:47.941 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 1 to smithi18136759-50 2024-09-17T16:02:47.942 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 2 to smithi18136759-50 2024-09-17T16:02:47.944 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 3 to smithi18136759-50 2024-09-17T16:02:47.945 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 4 to smithi18136759-50 2024-09-17T16:02:47.947 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 5 to smithi18136759-50 2024-09-17T16:02:47.948 INFO:tasks.rados.rados.0.smithi181.stdout:371: finishing write tid 6 to smithi18136759-50 2024-09-17T16:02:47.948 INFO:tasks.rados.rados.0.smithi181.stdout:371: oid 50 updating version 0 to 171 2024-09-17T16:02:47.948 INFO:tasks.rados.rados.0.smithi181.stdout:371: oid 50 version 171 is already newer than 169 2024-09-17T16:02:47.948 INFO:tasks.rados.rados.0.smithi181.stdout:371: oid 50 version 171 is already newer than 167 2024-09-17T16:02:47.949 INFO:tasks.rados.rados.0.smithi181.stdout:371: oid 50 version 171 is already newer than 168 2024-09-17T16:02:47.949 INFO:tasks.rados.rados.0.smithi181.stdout:371: oid 50 version 171 is already newer than 170 2024-09-17T16:02:47.949 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 50 v 171 (ObjNum 140 snap 42 seq_num 140) dirty exists 2024-09-17T16:02:47.949 INFO:tasks.rados.rados.0.smithi181.stdout:371: left oid 50 (ObjNum 140 snap 42 seq_num 140) 2024-09-17T16:02:47.968 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:47.969 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:47.969 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:47.969 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:47.969 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:47.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:47.986 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:47.985+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:48.086 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:48.086 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:48.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:48.086 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:248 3.cs0 3:3808462d:::smithi18136759-6:head [copy-from ver 2] snapc 9=[9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 65.154836779999997, 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019268496999999999, 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 248 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.087 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928582+0000", 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928583+0000", 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9899999999999999e-07 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928588+0000", 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2220000000000001e-06 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928601+0000", 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2717e-05 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928608+0000", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8600000000000004e-06 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928671+0000", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3791e-05 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928756+0000", 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4293000000000001e-05 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.940817+0000", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01206144 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.942933+0000", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021155280000000002 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947791+0000", 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048580639999999996 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.947850+0000", 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9682999999999998e-05 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:249 3.cs0 3:3808462d:::smithi18136759-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 65.154795668000006, 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013329483, 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.092 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 249 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928623+0000", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928624+0000", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2699999999999999e-07 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928626+0000", 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8940000000000002e-06 2024-09-17T16:02:48.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928643+0000", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6988999999999999e-05 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928651+0000", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.267e-06 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928901+0000", 2024-09-17T16:02:48.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025058200000000001 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.928955+0000", 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3973e-05 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941770+0000", 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012814828 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941852+0000", 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2083000000000007e-05 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:42.941953+0000", 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010034 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:285 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:48.097 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.151613349000002, 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010247877000000001, 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 285 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:48.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931805+0000", 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931808+0000", 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8779999999999998e-06 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931822+0000", 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3828e-05 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931850+0000", 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7500000000000001e-05 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931863+0000", 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3738e-05 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931969+0000", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001052 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931976+0000", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7300000000000005e-06 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938082+0000", 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061057769999999997 2024-09-17T16:02:48.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.938176+0000", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3714000000000004e-05 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939069+0000", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00089279400000000003 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941333+0000", 2024-09-17T16:02:48.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022647380000000001 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.941992+0000", 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00065853199999999998 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.942053+0000", 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1447999999999998e-05 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:286 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 2] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.151459658999997, 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013358431, 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 286 2024-09-17T16:02:48.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931959+0000", 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931960+0000", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000003e-07 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931965+0000", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9010000000000002e-06 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931974+0000", 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2520000000000005e-06 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931981+0000", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5950000000000004e-06 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932071+0000", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0390999999999996e-05 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932079+0000", 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2679999999999994e-06 2024-09-17T16:02:48.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939566+0000", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074865030000000003 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.939653+0000", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7336999999999994e-05 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940238+0000", 2024-09-17T16:02:48.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058448400000000002 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944090+0000", 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00385192 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945232+0000", 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001142784 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945317+0000", 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5080999999999994e-05 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:287 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc b=[b,a,9,8,7,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 62.151423719999997, 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013571456000000001, 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.110 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 287 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931995+0000", 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931996+0000", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1600000000000001e-07 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.931999+0000", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4300000000000002e-06 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932006+0000", 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2729999999999997e-06 2024-09-17T16:02:48.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932012+0000", 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8089999999999998e-06 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932107+0000", 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4751000000000001e-05 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:48.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.932112+0000", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6659999999999996e-06 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940481+0000", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083682089999999997 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940546+0000", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5704000000000007e-05 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.114 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.940923+0000", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00037686099999999998 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.944112+0000", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031884999999999999 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945503+0000", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001391226 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:45.945566+0000", 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3510999999999996e-05 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWriteReply(3.fs0 52/32 ECSubWriteReply(tid=51, last_complete=52'22, committed=1, applied=1))", 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.726447567000001, 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.138524809, 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.116 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356971+0000", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356972+0000", 2024-09-17T16:02:48.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.356976+0000", 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2950000000000004e-06 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495052+0000", 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13807575 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495065+0000", 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3276e-05 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495133+0000", 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8274000000000004e-05 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:50.495496+0000", 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036245899999999998 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.119 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:344 3.fs0 3:fd9ed431:::smithi18136759-28:head [stat,delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 55.747455637999998, 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012105121, 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 344 2024-09-17T16:02:48.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335963+0000", 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335966+0000", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.004e-06 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.335978+0000", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1810999999999999e-05 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336003+0000", 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5344e-05 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.122 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336017+0000", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3901e-05 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336125+0000", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107707 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.336136+0000", 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1476e-05 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340410+0000", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004273453 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.340525+0000", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115017 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.341450+0000", 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00092533800000000005 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.345523+0000", 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040730929999999999 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348008+0000", 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024844609999999999 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.348068+0000", 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0516000000000001e-05 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=129, reqid=osd.12.0:129, at_version=64'70, trim_to=0'0, roll_forward_to=62'66))", 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.743761980999999, 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081855320000000006, 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.126 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339657+0000", 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999998e-07 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339692+0000", 2024-09-17T16:02:48.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4459000000000003e-05 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339726+0000", 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4029000000000002e-05 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.339733+0000", 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0790000000000003e-06 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340271+0000", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00053823 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.340282+0000", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0606e-05 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347801+0000", 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075190999999999999 2024-09-17T16:02:48.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347842+0000", 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1387999999999997e-05 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.10s6 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.737855089, 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011676772, 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345564+0000", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.29e-07 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.345592+0000", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8580000000000001e-05 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346013+0000", 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00042032499999999998 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346021+0000", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2269999999999993e-06 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346353+0000", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033160900000000002 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346363+0000", 2024-09-17T16:02:48.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0390999999999999e-05 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357206+0000", 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010843148 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357240+0000", 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4263e-05 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 64/53 ECSubWrite(tid=44, reqid=osd.15.0:44, at_version=64'22, trim_to=0'0, roll_forward_to=63'20))", 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.736948036000001, 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010663848, 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:48.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346471+0000", 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7099999999999998e-07 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346489+0000", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7731999999999999e-05 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346664+0000", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017518199999999999 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346671+0000", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1239999999999999e-06 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346721+0000", 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.02e-05 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.346731+0000", 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.55e-06 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357095+0000", 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010364226000000001 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357135+0000", 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9563000000000001e-05 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.16s5 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.735568360000002, 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092773109999999999, 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.140 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347850+0000", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347851+0000", 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7400000000000002e-07 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347861+0000", 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0325e-05 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347893+0000", 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1173000000000001e-05 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347900+0000", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.306e-06 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347964+0000", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3678999999999998e-05 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.347973+0000", 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0790000000000007e-06 2024-09-17T16:02:48.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357093+0000", 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091199190000000006 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.357128+0000", 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5055999999999998e-05 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=131, reqid=osd.12.0:129, at_version=0'0, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.734292619999998, 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090524279999999995, 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349126+0000", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349127+0000", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349137+0000", 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0441999999999999e-05 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349186+0000", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8306999999999997e-05 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349195+0000", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3640000000000005e-06 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349414+0000", 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021941500000000001 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.349424+0000", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358144+0000", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087197060000000007 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.358179+0000", 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4814999999999998e-05 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.148 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds3 64/32 ECSubWrite(tid=133, reqid=osd.12.0:133, at_version=64'74, trim_to=0'0, roll_forward_to=64'70))", 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 45.730972227000002, 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079374270000000004, 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352446+0000", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352447+0000", 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352467+0000", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0146999999999999e-05 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352504+0000", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6365000000000002e-05 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352512+0000", 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0029999999999994e-06 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352565+0000", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3613999999999999e-05 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.352576+0000", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1141999999999999e-05 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360356+0000", 2024-09-17T16:02:48.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007779584 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:02.360384+0000", 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7847999999999998e-05 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s3 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:48.153 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 40.729997787000002, 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70314915600000005, 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353421+0000", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.54e-07 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353470+0000", 2024-09-17T16:02:48.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9177e-05 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055585+0000", 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70211433599999995 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055600+0000", 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4989e-05 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055790+0000", 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019022599999999999 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.055803+0000", 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3455000000000001e-05 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056540+0000", 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000736289 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:08.056570+0000", 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0530000000000001e-05 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:494 3.fs0 3:fd9ed431:::smithi18136759-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 40.709749189999997, 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082749039999999996, 2024-09-17T16:02:48.158 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 494 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373670+0000", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.51e-07 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373672+0000", 2024-09-17T16:02:48.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2689999999999998e-06 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373678+0000", 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2909999999999998e-06 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.373681+0000", 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3730000000000001e-06 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376047+0000", 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002366071 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.376106+0000", 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8610000000000003e-05 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381155+0000", 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050492460000000003 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381250+0000", 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5358000000000004e-05 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.381944+0000", 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069423499999999997 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s3 81/53 ECSubWrite(tid=74, reqid=osd.15.0:74, at_version=81'33, trim_to=0'0, roll_forward_to=77'31))", 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 28.656125961000001, 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0092459199999999995, 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427293+0000", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6199999999999999e-07 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427313+0000", 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0389000000000001e-05 2024-09-17T16:02:48.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427799+0000", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00048568899999999998 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.427805+0000", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1040000000000003e-06 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429581+0000", 2024-09-17T16:02:48.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017761179999999999 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.429592+0000", 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0621e-05 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436494+0000", 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069020920000000003 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436539+0000", 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4545000000000001e-05 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.6s5 85/65 ECSubWrite(tid=214, reqid=client.4560.0:753, at_version=85'24, trim_to=0'0, roll_forward_to=84'20))", 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 24.621147896, 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16302681799999999, 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.168 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462271+0000", 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462272+0000", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0200000000000001e-07 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.462292+0000", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0965999999999999e-05 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624643+0000", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.162350088 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624650+0000", 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4290000000000004e-06 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624728+0000", 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7996000000000003e-05 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.624749+0000", 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0990000000000001e-05 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625270+0000", 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052091300000000002 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:23.625298+0000", 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7733999999999999e-05 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:850 3.fs0 3:fe1a4813:::smithi18136759-15:head [rollback 16] snapc 29=[29,28,26,25,22,21,1c,19,18,16,c,6] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:02:48.172 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:41.519539+0000", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.5638792309999996, 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0077211550000000004, 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 850 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519539+0000", 2024-09-17T16:02:48.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519539+0000", 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519540+0000", 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0900000000000002e-07 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519543+0000", 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9890000000000001e-06 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519552+0000", 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8880000000000003e-06 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519558+0000", 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9240000000000004e-06 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519684+0000", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012602699999999999 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519695+0000", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0798999999999999e-05 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.523605+0000", 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039105110000000002 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.523723+0000", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011789 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.524395+0000", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00067200699999999996 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.526134+0000", 2024-09-17T16:02:48.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017386669999999999 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.527101+0000", 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00096690700000000003 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.527261+0000", 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000160037 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.4560.0:851 3.fs0 3:fe1a4813:::smithi18136759-15:head [append 0~0] snapc 29=[29,28,26,25,22,21,1c,19,18,16,c,6] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:41.519593+0000", 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 6.5638261660000001, 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0084821270000000008, 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.4560", 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:48.179 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 851 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519593+0000", 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519593+0000", 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519593+0000", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1500000000000005e-07 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519596+0000", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1130000000000001e-06 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519605+0000", 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:02:48.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519610+0000", 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.575e-06 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519716+0000", 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105766 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:48.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.519725+0000", 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0100000000000001e-06 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.524786+0000", 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00506107 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.524912+0000", 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012616299999999999 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.525487+0000", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000575083 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.526662+0000", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001174505 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.527948+0000", 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00128582 2024-09-17T16:02:48.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:41.528075+0000", 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127155 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es2 104/32 ECSubWrite(tid=156, reqid=client.4560.0:858, at_version=0'0, trim_to=0'0, roll_forward_to=104'143))", 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:42.546262+0000", 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 5.537156553, 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.74477155100000003, 2024-09-17T16:02:48.185 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:42.546262+0000", 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:42.546262+0000", 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:42.546264+0000", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.514e-06 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:42.546279+0000", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5586e-05 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.290374+0000", 2024-09-17T16:02:48.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.74409508400000002 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.290384+0000", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8099999999999992e-06 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.290462+0000", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7750999999999995e-05 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.290475+0000", 2024-09-17T16:02:48.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3458999999999999e-05 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.290986+0000", 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051018600000000004 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:43.291034+0000", 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8161000000000002e-05 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.190 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.190 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:48.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:48.192 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:48.189+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:48.291 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:48.289+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:48.340 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:48.341 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:48.341 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:48.341 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:48.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:48.392 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:48.391+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:48.449 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:48.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:48.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:02:48.493 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:48.491+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:48.496 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:48.497 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:48.497 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:48.497 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:48.497 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:48.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:48.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:48.563 INFO:tasks.rados.rados.0.smithi181.stdout:370: done (2 left) 2024-09-17T16:02:48.564 INFO:tasks.rados.rados.0.smithi181.stdout:371: done (1 left) 2024-09-17T16:02:48.564 INFO:tasks.rados.rados.0.smithi181.stdout:372: done (0 left) 2024-09-17T16:02:48.564 INFO:tasks.rados.rados.0.smithi181.stdout:373: setattr oid 6 current snap is 43 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:374: copy_from oid 47 from oid 20 current snap is 43 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:375: read oid 4 snap -1 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:375: expect (ObjNum 120 snap 29 seq_num 120) 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:376: read oid 15 snap 23 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:376: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T16:02:48.565 INFO:tasks.rados.rados.0.smithi181.stdout:377: snap_create 2024-09-17T16:02:48.574 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 60 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:02:48.575 INFO:tasks.rados.rados.0.smithi181.stdout:374: finishing copy_from to smithi18136759-47 2024-09-17T16:02:48.575 INFO:tasks.rados.rados.0.smithi181.stdout:374: got expected ENOENT (src dne) 2024-09-17T16:02:48.575 INFO:tasks.rados.rados.0.smithi181.stdout:374: finishing copy_from racing read to smithi18136759-47 2024-09-17T16:02:48.575 INFO:tasks.rados.rados.0.smithi181.stdout:376: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T16:02:48.576 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:48.578 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:48.580 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:48.582 INFO:tasks.rados.rados.0.smithi181.stdout:375: expect (ObjNum 120 snap 29 seq_num 120) 2024-09-17T16:02:48.592 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 1845453 2024-09-17T16:02:48.594 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:48.591+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:48.625 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=182, reqid=osd.11.0:182, at_version=46'20, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 63.705333173, 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066826129999999996, 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917859+0000", 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5800000000000001e-07 2024-09-17T16:02:48.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.917886+0000", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6931999999999999e-05 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918192+0000", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030561400000000001 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918355+0000", 2024-09-17T16:02:48.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016328600000000001 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918431+0000", 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5690000000000002e-05 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918441+0000", 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9450000000000005e-06 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924513+0000", 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060725709999999997 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924542+0000", 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8317e-05 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 46/32 ECSubWrite(tid=45, reqid=osd.14.0:45, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:48.630 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 63.704654230000003, 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0060611290000000002, 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:48.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918538+0000", 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.96e-07 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918568+0000", 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0068000000000001e-05 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918670+0000", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010144 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918678+0000", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0900000000000005e-06 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918750+0000", 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2100000000000004e-05 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918761+0000", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0767999999999999e-05 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924554+0000", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057936560000000003 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924599+0000", 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4710999999999999e-05 2024-09-17T16:02:48.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=46, reqid=osd.14.0:46, at_version=46'17, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 63.704617130000003, 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059181870000000001, 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918575+0000", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918576+0000", 2024-09-17T16:02:48.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8400000000000005e-07 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918591+0000", 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5583999999999999e-05 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918817+0000", 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022566799999999999 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918825+0000", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9389999999999997e-06 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918894+0000", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9332000000000006e-05 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.918904+0000", 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9860000000000006e-06 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924448+0000", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055442470000000004 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.924493+0000", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4947000000000002e-05 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.2s3 46/32 ECSubWrite(tid=184, reqid=osd.11.0:184, at_version=46'22, trim_to=0'0, roll_forward_to=45'18))", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:48.639 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 63.703644519999997, 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056807560000000003, 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919548+0000", 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919549+0000", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7900000000000007e-07 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919571+0000", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2155999999999999e-05 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919754+0000", 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018332299999999999 2024-09-17T16:02:48.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919767+0000", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2656e-05 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920277+0000", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051049799999999996 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920289+0000", 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1388e-05 2024-09-17T16:02:48.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925184+0000", 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048955539999999999 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925228+0000", 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4202000000000001e-05 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.643 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.0s2 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 63.703604736000003, 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056906200000000004, 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919588+0000", 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1299999999999999e-07 2024-09-17T16:02:48.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919607+0000", 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9062000000000001e-05 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919796+0000", 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018866 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.919803+0000", 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4150000000000002e-06 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920325+0000", 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052165800000000004 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.920334+0000", 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1770000000000004e-06 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925242+0000", 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049072009999999999 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:44.925278+0000", 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6634000000000002e-05 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.648 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 54/32 ECSubWrite(tid=96, reqid=client.4560.0:345, at_version=0'0, trim_to=0'0, roll_forward_to=54'43))", 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.264430615999999, 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067241940000000002, 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358762+0000", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358763+0000", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1710000000000001e-06 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358778+0000", 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5158e-05 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358817+0000", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8832999999999999e-05 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358826+0000", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2390000000000008e-06 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358909+0000", 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2556e-05 2024-09-17T16:02:48.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.358920+0000", 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0910000000000001e-05 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365448+0000", 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006528748 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365486+0000", 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7579000000000001e-05 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 54/53 ECSubWrite(tid=23, reqid=client.4560.0:346, at_version=54'18, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 56.263695646999999, 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059831219999999996, 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.653 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359497+0000", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359521+0000", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3705000000000001e-05 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359561+0000", 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9879999999999998e-05 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359570+0000", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1640000000000007e-06 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359645+0000", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4765000000000007e-05 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.359656+0000", 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0702e-05 2024-09-17T16:02:48.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365440+0000", 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0057839320000000003 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:01:52.365480+0000", 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0219000000000002e-05 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 69/32 ECSubWrite(tid=70, reqid=osd.2.0:70, at_version=69'51, trim_to=0'0, roll_forward_to=68'49))", 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 41.284348297999998, 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24156233099999999, 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338844+0000", 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3900000000000001e-07 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.338868+0000", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3495000000000001e-05 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579761+0000", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.24089318000000001 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579766+0000", 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579834+0000", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7265000000000001e-05 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.579841+0000", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.029e-06 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580381+0000", 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000540626 2024-09-17T16:02:48.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.580406+0000", 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4958999999999999e-05 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.13s5 69/53 ECSubWrite(tid=54, reqid=osd.15.0:54, at_version=69'28, trim_to=0'0, roll_forward_to=68'26))", 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 41.269851731999999, 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32307851999999998, 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.662 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353341+0000", 2024-09-17T16:02:48.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6999999999999999e-07 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.353366+0000", 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5431000000000001e-05 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675724+0000", 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32235754599999999 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675729+0000", 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1100000000000002e-06 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675814+0000", 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.551e-05 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.675822+0000", 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676394+0000", 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057224999999999995 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:07.676419+0000", 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5049e-05 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.666 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=93, reqid=client.4560.0:634, at_version=79'76, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 31.191141472999998, 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068757990000000001, 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432051+0000", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432072+0000", 2024-09-17T16:02:48.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0971e-05 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434134+0000", 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002062342 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434144+0000", 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0039e-05 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434203+0000", 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8757e-05 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434214+0000", 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0879e-05 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438888+0000", 2024-09-17T16:02:48.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046737510000000003 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438927+0000", 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8779000000000003e-05 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/32 ECSubWrite(tid=226, reqid=client.4560.0:628, at_version=79'153, trim_to=0'0, roll_forward_to=78'147))", 2024-09-17T16:02:48.672 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 31.190248449999999, 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059654479999999999, 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432944+0000", 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432945+0000", 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.432987+0000", 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2401999999999998e-05 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433023+0000", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5562999999999998e-05 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433033+0000", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0539e-05 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433334+0000", 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000300952 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.433343+0000", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.302e-06 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438871+0000", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055279500000000002 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.438909+0000", 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7951999999999998e-05 2024-09-17T16:02:48.676 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=94, reqid=osd.2.0:94, at_version=79'78, trim_to=0'0, roll_forward_to=78'74))", 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 31.189096133, 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079659940000000005, 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.677 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434096+0000", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434097+0000", 2024-09-17T16:02:48.678 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.434118+0000", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0834000000000001e-05 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440520+0000", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064023750000000001 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440528+0000", 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6120000000000001e-06 2024-09-17T16:02:48.679 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440597+0000", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9429000000000002e-05 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.440608+0000", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1199000000000001e-05 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442024+0000", 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001415166 2024-09-17T16:02:48.680 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.442062+0000", 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8469e-05 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 79/32 ECSubWrite(tid=98, reqid=client.4560.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 31.175337141, 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25686310400000001, 2024-09-17T16:02:48.681 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447855+0000", 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.682 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447856+0000", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.71e-07 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.447869+0000", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2795e-05 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703761+0000", 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25589224300000002 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.683 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703768+0000", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6599999999999998e-06 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703842+0000", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4402999999999999e-05 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.703857+0000", 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4737e-05 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.684 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704682+0000", 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00082487599999999999 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:17.704718+0000", 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6519000000000001e-05 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.685 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=99, reqid=osd.2.0:99, at_version=81'83, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.193183622999999, 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066388380000000002, 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.686 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430009+0000", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0100000000000001e-07 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430030+0000", 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1211000000000001e-05 2024-09-17T16:02:48.687 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430055+0000", 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5411000000000001e-05 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430066+0000", 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0635000000000001e-05 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.688 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430142+0000", 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5414999999999995e-05 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.430154+0000", 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2877000000000001e-05 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.689 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436610+0000", 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064551849999999996 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436647+0000", 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7902999999999999e-05 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 81/32 ECSubWrite(tid=100, reqid=osd.2.0:100, at_version=81'85, trim_to=0'0, roll_forward_to=79'81))", 2024-09-17T16:02:48.690 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.192183443000001, 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056630860000000003, 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.691 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431009+0000", 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431010+0000", 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3399999999999997e-07 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431035+0000", 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5491999999999999e-05 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.692 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431082+0000", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6853e-05 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431092+0000", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0164e-05 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431183+0000", 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1129000000000001e-05 2024-09-17T16:02:48.693 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.431195+0000", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1191e-05 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436653+0000", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054587730000000001 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.436672+0000", 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.855e-05 2024-09-17T16:02:48.694 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.19s4 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.174599369999999, 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063649290000000001, 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:48.695 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448593+0000", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448594+0000", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3799999999999996e-07 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448861+0000", 2024-09-17T16:02:48.696 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026757000000000001 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448891+0000", 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9943000000000001e-05 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.448897+0000", 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.823e-06 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.697 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449029+0000", 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013207599999999999 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.449043+0000", 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.414e-05 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454916+0000", 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058721440000000001 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.698 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.454958+0000", 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2295000000000001e-05 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.9s4 81/53 ECSubWrite(tid=155, reqid=osd.14.0:155, at_version=81'47, trim_to=0'0, roll_forward_to=77'43))", 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 29.172179490000001, 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065741319999999999, 2024-09-17T16:02:48.699 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451013+0000", 2024-09-17T16:02:48.700 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451014+0000", 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451051+0000", 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7224999999999997e-05 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.701 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451087+0000", 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5895000000000001e-05 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451097+0000", 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0706e-05 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.702 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451168+0000", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0992000000000001e-05 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.451179+0000", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0185e-05 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457549+0000", 2024-09-17T16:02:48.703 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063700420000000002 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:19.457587+0000", 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8297000000000003e-05 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.es3 93/32 ECSubWrite(tid=133, reqid=osd.2.0:133, at_version=93'117, trim_to=0'0, roll_forward_to=92'115))", 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:48.704 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.116537945000001, 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059325740000000004, 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.705 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506654+0000", 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506655+0000", 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.16e-07 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506677+0000", 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2242000000000002e-05 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.706 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506717+0000", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0519999999999998e-05 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506726+0000", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3329999999999997e-06 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.506996+0000", 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027085500000000002 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.707 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507008+0000", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1080999999999999e-05 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512549+0000", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055410959999999997 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.512587+0000", 2024-09-17T16:02:48.708 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8231e-05 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 93/53 ECSubWrite(tid=89, reqid=osd.15.0:89, at_version=93'41, trim_to=0'0, roll_forward_to=84'39))", 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 17.115318218999999, 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076683949999999997, 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.709 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507874+0000", 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.49e-07 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.710 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507894+0000", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9981000000000001e-05 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507975+0000", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1248999999999994e-05 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.507984+0000", 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2090000000000005e-06 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.711 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508051+0000", 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6963000000000002e-05 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.508062+0000", 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1418e-05 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.712 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515497+0000", 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0074346079999999997 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:31.515542+0000", 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5818000000000001e-05 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.713 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDECSubOpWrite(3.3s5 98/53 ECSubWrite(tid=92, reqid=osd.15.0:91, at_version=0'0, trim_to=0'0, roll_forward_to=98'43))", 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 12.104543194, 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00573932, 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:48.714 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518649+0000", 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518650+0000", 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6300000000000005e-07 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.715 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518670+0000", 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0757000000000001e-05 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518703+0000", 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2963999999999998e-05 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-09-17T16:02:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518712+0000", 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4710000000000007e-06 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518782+0000", 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0405000000000001e-05 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_started", 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.518793+0000", 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.045e-05 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.717 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_committed", 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.524353+0000", 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055602189999999999 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-09-17T16:02:36.524388+0000", 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5391e-05 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-09-17T16:02:48.718 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:48.720 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 3296760 2024-09-17T16:02:48.720 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:48.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:48.721 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:48.718+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:48.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:48.782 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:48.784 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:02:48.798 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-17T16:02:48.558016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122831411,"num_objects":165,"num_object_clones":111,"num_object_copies":1135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":10,"num_read":376,"num_read_kb":116222,"num_write":688,"num_write_kb":107826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1634,"ondisk_log_size":1634,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":709192,"kb_used_data":218436,"kb_used_omap":24,"kb_used_meta":490343,"kb_avail":1509240248,"statfs":{"total":1546188226560,"available":1545462013952,"internally_reserved":0,"allocated":223678464,"data_stored":328727780,"data_compressed":3981044,"data_compressed_allocated":5353472,"data_compressed_original":11583488,"omap_allocated":25519,"internal_metadata":502111313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3817472,"num_objects":3,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":19991,"num_write":52,"num_write_kb":13024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.025391"},"pg_stats":[{"pgid":"3.19","version":"106'59","reported_seq":479,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.543244+0000","last_change":"2024-09-17T16:02:44.544197+0000","last_active":"2024-09-17T16:02:44.543244+0000","last_peered":"2024-09-17T16:02:44.543244+0000","last_clean":"2024-09-17T16:02:44.543244+0000","last_became_active":"2024-09-17T16:02:36.523148+0000","last_became_peered":"2024-09-17T16:02:36.523148+0000","last_unstale":"2024-09-17T16:02:44.543244+0000","last_undegraded":"2024-09-17T16:02:44.543244+0000","last_fullsized":"2024-09-17T16:02:44.543244+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:01:59.252035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.252035+0000","objects_scrubbed":4,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:28.506298+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0057792870000000001,"stat_sum":{"num_bytes":6275072,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":4719,"num_write":10,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8945664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,1],"acting":[14,11,13,15,4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.18","version":"100'66","reported_seq":459,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.542565+0000","last_change":"2024-09-17T16:02:44.543821+0000","last_active":"2024-09-17T16:02:44.542565+0000","last_peered":"2024-09-17T16:02:44.542565+0000","last_clean":"2024-09-17T16:02:44.542565+0000","last_became_active":"2024-09-17T16:01:53.177881+0000","last_became_peered":"2024-09-17T16:01:53.177881+0000","last_unstale":"2024-09-17T16:02:44.542565+0000","last_undegraded":"2024-09-17T16:02:44.542565+0000","last_fullsized":"2024-09-17T16:02:44.542565+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'31","last_scrub_stamp":"2024-09-17T16:02:00.137409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.137409+0000","objects_scrubbed":2,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:01.233537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001595178,"stat_sum":{"num_bytes":5603328,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":19990,"num_write":13,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.17","version":"84'44","reported_seq":350,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534392+0000","last_change":"2024-09-17T16:02:04.352378+0000","last_active":"2024-09-17T16:02:44.534392+0000","last_peered":"2024-09-17T16:02:44.534392+0000","last_clean":"2024-09-17T16:02:02.351935+0000","last_became_active":"2024-09-17T16:02:04.352378+0000","last_became_peered":"2024-09-17T16:02:04.352378+0000","last_unstale":"2024-09-17T16:02:44.534392+0000","last_undegraded":"2024-09-17T16:02:04.343159+0000","last_fullsized":"2024-09-17T16:02:04.342807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'27","last_scrub_stamp":"2024-09-17T16:02:02.077678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.077678+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:22.979962+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.027309317,"stat_sum":{"num_bytes":5914624,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":6628,"num_write":11,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.16","version":"88'38","reported_seq":351,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534072+0000","last_change":"2024-09-17T16:02:04.351696+0000","last_active":"2024-09-17T16:02:44.534072+0000","last_peered":"2024-09-17T16:02:44.534072+0000","last_clean":"2024-09-17T16:02:02.354687+0000","last_became_active":"2024-09-17T16:02:04.351696+0000","last_became_peered":"2024-09-17T16:02:04.351696+0000","last_unstale":"2024-09-17T16:02:44.534072+0000","last_undegraded":"2024-09-17T16:02:04.341016+0000","last_fullsized":"2024-09-17T16:02:04.340608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'16","last_scrub_stamp":"2024-09-17T16:02:00.050173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.050173+0000","objects_scrubbed":4,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:28.214553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01423026,"stat_sum":{"num_bytes":6881280,"num_objects":9,"num_object_clones":7,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":8,"num_read_kb":7052,"num_write":12,"num_write_kb":3232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":9}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.15","version":"106'38","reported_seq":430,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.549976+0000","last_change":"2024-09-17T16:02:44.551129+0000","last_active":"2024-09-17T16:02:44.549976+0000","last_peered":"2024-09-17T16:02:44.549976+0000","last_clean":"2024-09-17T16:02:44.549976+0000","last_became_active":"2024-09-17T16:01:53.142297+0000","last_became_peered":"2024-09-17T16:01:53.142297+0000","last_unstale":"2024-09-17T16:02:44.549976+0000","last_undegraded":"2024-09-17T16:02:44.549976+0000","last_fullsized":"2024-09-17T16:02:44.549976+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'8","last_scrub_stamp":"2024-09-17T16:01:58.940426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.940426+0000","objects_scrubbed":2,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:21.941546+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0.0096733400000000008,"stat_sum":{"num_bytes":2834432,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.14","version":"100'34","reported_seq":422,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T16:02:43.598490+0000","last_change":"2024-09-17T16:02:43.599080+0000","last_active":"2024-09-17T16:02:43.598490+0000","last_peered":"2024-09-17T16:02:43.598490+0000","last_clean":"2024-09-17T16:02:43.598490+0000","last_became_active":"2024-09-17T16:01:52.360122+0000","last_became_peered":"2024-09-17T16:01:52.360122+0000","last_unstale":"2024-09-17T16:02:43.598490+0000","last_undegraded":"2024-09-17T16:02:43.598490+0000","last_fullsized":"2024-09-17T16:02:43.598490+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:58.496684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.496684+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:03.831239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094379200000000005,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"},{"start":"1d","length":"5"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":102,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.534217+0000","last_change":"2024-09-17T16:01:58.347762+0000","last_active":"2024-09-17T16:02:44.534217+0000","last_peered":"2024-09-17T16:02:44.534217+0000","last_clean":"2024-09-17T16:02:44.534217+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:02:44.534217+0000","last_undegraded":"2024-09-17T16:02:44.534217+0000","last_fullsized":"2024-09-17T16:02:44.534217+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:34.031282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"85'24","reported_seq":330,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534889+0000","last_change":"2024-09-17T16:02:04.349120+0000","last_active":"2024-09-17T16:02:44.534889+0000","last_peered":"2024-09-17T16:02:44.534889+0000","last_clean":"2024-09-17T16:02:02.355614+0000","last_became_active":"2024-09-17T16:02:04.349120+0000","last_became_peered":"2024-09-17T16:02:04.349120+0000","last_unstale":"2024-09-17T16:02:44.534889+0000","last_undegraded":"2024-09-17T16:02:04.340835+0000","last_fullsized":"2024-09-17T16:02:04.340587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:02:01.398192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.398192+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:11.585674+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0.018760373,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":964,"num_write":9,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.542640+0000","last_change":"2024-09-17T16:01:20.769736+0000","last_active":"2024-09-17T16:02:44.542640+0000","last_peered":"2024-09-17T16:02:44.542640+0000","last_clean":"2024-09-17T16:02:44.542640+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:02:44.542640+0000","last_undegraded":"2024-09-17T16:02:44.542640+0000","last_fullsized":"2024-09-17T16:02:44.542640+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:46.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"72'22","reported_seq":314,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534576+0000","last_change":"2024-09-17T16:02:04.349605+0000","last_active":"2024-09-17T16:02:44.534576+0000","last_peered":"2024-09-17T16:02:44.534576+0000","last_clean":"2024-09-17T16:02:03.036314+0000","last_became_active":"2024-09-17T16:02:04.349605+0000","last_became_peered":"2024-09-17T16:02:04.349605+0000","last_unstale":"2024-09-17T16:02:44.534576+0000","last_undegraded":"2024-09-17T16:02:04.340916+0000","last_fullsized":"2024-09-17T16:02:04.340610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01416717,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":15,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.541927+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:02:44.541927+0000","last_peered":"2024-09-17T16:02:44.541927+0000","last_clean":"2024-09-17T16:02:44.541927+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:02:44.541927+0000","last_undegraded":"2024-09-17T16:02:44.541927+0000","last_fullsized":"2024-09-17T16:02:44.541927+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:19.517195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":391,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T16:02:43.598871+0000","last_change":"2024-09-17T16:02:43.599497+0000","last_active":"2024-09-17T16:02:43.598871+0000","last_peered":"2024-09-17T16:02:43.598871+0000","last_clean":"2024-09-17T16:02:43.598871+0000","last_became_active":"2024-09-17T16:01:52.351079+0000","last_became_peered":"2024-09-17T16:01:52.351079+0000","last_unstale":"2024-09-17T16:02:43.598871+0000","last_undegraded":"2024-09-17T16:02:43.598871+0000","last_fullsized":"2024-09-17T16:02:43.598871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:59.453974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.453974+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:13.680755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000925445,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"2"},{"start":"1d","length":"5"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":180,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T16:02:43.598439+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:02:43.598439+0000","last_peered":"2024-09-17T16:02:43.598439+0000","last_clean":"2024-09-17T16:02:43.598439+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:02:43.598439+0000","last_undegraded":"2024-09-17T16:02:43.598439+0000","last_fullsized":"2024-09-17T16:02:43.598439+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:43.317628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"106'43","reported_seq":428,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.553787+0000","last_change":"2024-09-17T16:02:44.555208+0000","last_active":"2024-09-17T16:02:44.553787+0000","last_peered":"2024-09-17T16:02:44.553787+0000","last_clean":"2024-09-17T16:02:44.553787+0000","last_became_active":"2024-09-17T16:01:52.345743+0000","last_became_peered":"2024-09-17T16:01:52.345743+0000","last_unstale":"2024-09-17T16:02:44.553787+0000","last_undegraded":"2024-09-17T16:02:44.553787+0000","last_fullsized":"2024-09-17T16:02:44.553787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'8","last_scrub_stamp":"2024-09-17T16:01:59.457065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.457065+0000","objects_scrubbed":2,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:01.520697+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013601047,"stat_sum":{"num_bytes":344064,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.3","version":"106'47","reported_seq":438,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.548204+0000","last_change":"2024-09-17T16:02:44.549679+0000","last_active":"2024-09-17T16:02:44.548204+0000","last_peered":"2024-09-17T16:02:44.548204+0000","last_clean":"2024-09-17T16:02:44.548204+0000","last_became_active":"2024-09-17T16:01:52.349931+0000","last_became_peered":"2024-09-17T16:01:52.349931+0000","last_unstale":"2024-09-17T16:02:44.548204+0000","last_undegraded":"2024-09-17T16:02:44.548204+0000","last_fullsized":"2024-09-17T16:02:44.548204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'18","last_scrub_stamp":"2024-09-17T16:01:59.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.099258+0000","objects_scrubbed":4,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:22.144130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0068495079999999998,"stat_sum":{"num_bytes":1212416,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":1605,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":186,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.541306+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:02:44.541306+0000","last_peered":"2024-09-17T16:02:44.541306+0000","last_clean":"2024-09-17T16:02:44.541306+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:02:44.541306+0000","last_undegraded":"2024-09-17T16:02:44.541306+0000","last_fullsized":"2024-09-17T16:02:44.541306+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"106'111","reported_seq":487,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.579483+0000","last_change":"2024-09-17T16:02:44.561971+0000","last_active":"2024-09-17T16:02:44.579483+0000","last_peered":"2024-09-17T16:02:44.579483+0000","last_clean":"2024-09-17T16:02:44.579483+0000","last_became_active":"2024-09-17T16:01:52.345493+0000","last_became_peered":"2024-09-17T16:01:52.345493+0000","last_unstale":"2024-09-17T16:02:44.579483+0000","last_undegraded":"2024-09-17T16:02:44.579483+0000","last_fullsized":"2024-09-17T16:02:44.579483+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'27","last_scrub_stamp":"2024-09-17T16:02:00.289587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.289587+0000","objects_scrubbed":4,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:00.654008+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.023433243999999999,"stat_sum":{"num_bytes":7077888,"num_objects":10,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":12,"num_read_kb":3111,"num_write":13,"num_write_kb":5776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.541494+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:02:44.541494+0000","last_peered":"2024-09-17T16:02:44.541494+0000","last_clean":"2024-09-17T16:02:44.541494+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:02:44.541494+0000","last_undegraded":"2024-09-17T16:02:44.541494+0000","last_fullsized":"2024-09-17T16:02:44.541494+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:41.693243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"100'23","reported_seq":328,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534013+0000","last_change":"2024-09-17T16:02:04.352734+0000","last_active":"2024-09-17T16:02:44.534013+0000","last_peered":"2024-09-17T16:02:44.534013+0000","last_clean":"2024-09-17T16:02:02.350917+0000","last_became_active":"2024-09-17T16:02:04.352734+0000","last_became_peered":"2024-09-17T16:02:04.352734+0000","last_unstale":"2024-09-17T16:02:44.534013+0000","last_undegraded":"2024-09-17T16:02:04.341137+0000","last_fullsized":"2024-09-17T16:02:04.340823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.086155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.086155+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:25.924120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014571756,"stat_sum":{"num_bytes":2260992,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":447,"num_write":24,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,2147483647],"acting":[13,4,3,11,10,2,2147483647],"avail_no_missing":["13(0)","2(5)","3(2)","4(1)","10(4)","11(3)"],"object_location_counts":[{"shards":"2(5),3(2),4(1),10(4),11(3),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.541113+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:02:44.541113+0000","last_peered":"2024-09-17T16:02:44.541113+0000","last_clean":"2024-09-17T16:02:44.541113+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:02:44.541113+0000","last_undegraded":"2024-09-17T16:02:44.541113+0000","last_fullsized":"2024-09-17T16:02:44.541113+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"97'49","reported_seq":352,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.542253+0000","last_change":"2024-09-17T16:02:04.349933+0000","last_active":"2024-09-17T16:02:44.542253+0000","last_peered":"2024-09-17T16:02:44.542253+0000","last_clean":"2024-09-17T16:02:02.354956+0000","last_became_active":"2024-09-17T16:02:04.349933+0000","last_became_peered":"2024-09-17T16:02:04.349933+0000","last_unstale":"2024-09-17T16:02:44.542253+0000","last_undegraded":"2024-09-17T16:02:04.341956+0000","last_fullsized":"2024-09-17T16:02:04.341160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'35","last_scrub_stamp":"2024-09-17T16:02:01.159864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.159864+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:02.246299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023769390000000001,"stat_sum":{"num_bytes":4030464,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":4,"num_read_kb":138,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,2147483647],"acting":[11,2,0,4,8,3,2147483647],"avail_no_missing":["11(0)","0(2)","2(1)","3(5)","4(3)","8(4)"],"object_location_counts":[{"shards":"0(2),2(1),3(5),4(3),8(4),11(0)","objects":11}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":176,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.542888+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:02:44.542888+0000","last_peered":"2024-09-17T16:02:44.542888+0000","last_clean":"2024-09-17T16:02:44.542888+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:02:44.542888+0000","last_undegraded":"2024-09-17T16:02:44.542888+0000","last_fullsized":"2024-09-17T16:02:44.542888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:41.299345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":318,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.543063+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:02:44.543063+0000","last_peered":"2024-09-17T16:02:44.543063+0000","last_clean":"2024-09-17T16:02:44.543063+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:02:44.543063+0000","last_undegraded":"2024-09-17T16:02:44.543063+0000","last_fullsized":"2024-09-17T16:02:44.543063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:42.104346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"106'62","reported_seq":444,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.552375+0000","last_change":"2024-09-17T16:02:44.554239+0000","last_active":"2024-09-17T16:02:44.552375+0000","last_peered":"2024-09-17T16:02:44.552375+0000","last_clean":"2024-09-17T16:02:44.552375+0000","last_became_active":"2024-09-17T16:01:52.347600+0000","last_became_peered":"2024-09-17T16:01:52.347600+0000","last_unstale":"2024-09-17T16:02:44.552375+0000","last_undegraded":"2024-09-17T16:02:44.552375+0000","last_fullsized":"2024-09-17T16:02:44.552375+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'33","last_scrub_stamp":"2024-09-17T16:02:02.109565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.109565+0000","objects_scrubbed":3,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:22.586476+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.011310641,"stat_sum":{"num_bytes":2277376,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":10,"num_read_kb":10525,"num_write":7,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.9","version":"106'73","reported_seq":476,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.546938+0000","last_change":"2024-09-17T16:02:44.548687+0000","last_active":"2024-09-17T16:02:44.546938+0000","last_peered":"2024-09-17T16:02:44.546938+0000","last_clean":"2024-09-17T16:02:44.546938+0000","last_became_active":"2024-09-17T16:02:36.521024+0000","last_became_peered":"2024-09-17T16:02:36.521024+0000","last_unstale":"2024-09-17T16:02:44.546938+0000","last_undegraded":"2024-09-17T16:02:44.546938+0000","last_fullsized":"2024-09-17T16:02:44.546938+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:02:01.251580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.251580+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:08.071230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010277906,"stat_sum":{"num_bytes":5128192,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":452,"num_write":9,"num_write_kb":4000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5373952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,1],"acting":[14,11,13,15,4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.a","version":"107'65","reported_seq":345,"reported_epoch":107,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:45.562489+0000","last_change":"2024-09-17T16:02:04.994580+0000","last_active":"2024-09-17T16:02:45.562489+0000","last_peered":"2024-09-17T16:02:45.562489+0000","last_clean":"2024-09-17T16:02:02.349037+0000","last_became_active":"2024-09-17T16:02:04.994580+0000","last_became_peered":"2024-09-17T16:02:04.994580+0000","last_unstale":"2024-09-17T16:02:45.562489+0000","last_undegraded":"2024-09-17T16:02:04.341211+0000","last_fullsized":"2024-09-17T16:02:04.340891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081311999999999,"stat_sum":{"num_bytes":12795904,"num_objects":11,"num_object_clones":9,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":19,"num_read_kb":20812,"num_write":35,"num_write_kb":14704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,2147483647,3,14,6,8],"acting":[12,4,2147483647,3,14,6,8],"avail_no_missing":["12(0)","3(3)","4(1)","6(5)","8(6)","14(4)"],"object_location_counts":[{"shards":"3(3),4(1),6(5),8(6),12(0),14(4)","objects":11}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.b","version":"84'24","reported_seq":312,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.533919+0000","last_change":"2024-09-17T16:02:04.347340+0000","last_active":"2024-09-17T16:02:44.533919+0000","last_peered":"2024-09-17T16:02:44.533919+0000","last_clean":"2024-09-17T16:02:02.337229+0000","last_became_active":"2024-09-17T16:02:04.347340+0000","last_became_peered":"2024-09-17T16:02:04.347340+0000","last_unstale":"2024-09-17T16:02:44.533919+0000","last_undegraded":"2024-09-17T16:02:04.340875+0000","last_fullsized":"2024-09-17T16:02:04.340555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:44.970304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010428939999999999,"stat_sum":{"num_bytes":3063808,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":21,"num_read_kb":1683,"num_write":13,"num_write_kb":2992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,2147483647],"acting":[13,1,11,15,6,4,2147483647],"avail_no_missing":["13(0)","1(1)","4(5)","6(4)","11(2)","15(3)"],"object_location_counts":[{"shards":"1(1),4(5),6(4),11(2),13(0),15(3)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.c","version":"100'58","reported_seq":419,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.539636+0000","last_change":"2024-09-17T16:02:44.540121+0000","last_active":"2024-09-17T16:02:44.539636+0000","last_peered":"2024-09-17T16:02:44.539636+0000","last_clean":"2024-09-17T16:02:44.539636+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:02:44.539636+0000","last_undegraded":"2024-09-17T16:02:44.539636+0000","last_fullsized":"2024-09-17T16:02:44.539636+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'56","last_scrub_stamp":"2024-09-17T16:02:35.300544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.300544+0000","objects_scrubbed":5,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:36.561045+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0012742140000000001,"stat_sum":{"num_bytes":3473408,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":6104,"num_write":37,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.d","version":"107'263","reported_seq":618,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T16:02:45.576121+0000","last_change":"2024-09-17T16:02:45.577012+0000","last_active":"2024-09-17T16:02:45.576121+0000","last_peered":"2024-09-17T16:02:45.576121+0000","last_clean":"2024-09-17T16:02:45.576121+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:02:45.576121+0000","last_undegraded":"2024-09-17T16:02:45.576121+0000","last_fullsized":"2024-09-17T16:02:45.576121+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025281254999999999,"stat_sum":{"num_bytes":14647296,"num_objects":18,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":59,"num_read_kb":18326,"num_write":96,"num_write_kb":24496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.e","version":"106'161","reported_seq":518,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.568400+0000","last_change":"2024-09-17T16:02:44.564815+0000","last_active":"2024-09-17T16:02:44.568400+0000","last_peered":"2024-09-17T16:02:44.568400+0000","last_clean":"2024-09-17T16:02:44.568400+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:02:44.568400+0000","last_undegraded":"2024-09-17T16:02:44.568400+0000","last_fullsized":"2024-09-17T16:02:44.568400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016355931000000001,"stat_sum":{"num_bytes":18268160,"num_objects":17,"num_object_clones":13,"num_object_copies":119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":37,"num_read_kb":11723,"num_write":100,"num_write_kb":15424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.f","version":"103'47","reported_seq":319,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.539539+0000","last_change":"2024-09-17T16:02:04.347090+0000","last_active":"2024-09-17T16:02:44.539539+0000","last_peered":"2024-09-17T16:02:44.539539+0000","last_clean":"2024-09-17T16:02:02.358329+0000","last_became_active":"2024-09-17T16:02:04.347090+0000","last_became_peered":"2024-09-17T16:02:04.347090+0000","last_unstale":"2024-09-17T16:02:44.539539+0000","last_undegraded":"2024-09-17T16:02:04.341882+0000","last_fullsized":"2024-09-17T16:02:04.341582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:48.967713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015713103999999999,"stat_sum":{"num_bytes":2605056,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":17,"num_read_kb":462,"num_write":40,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,2147483647,10,1],"acting":[0,2,14,11,2147483647,10,1],"avail_no_missing":["0(0)","1(6)","2(1)","10(5)","11(3)","14(2)"],"object_location_counts":[{"shards":"0(0),1(6),2(1),10(5),11(3),14(2)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.10","version":"106'49","reported_seq":233,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.579626+0000","last_change":"2024-09-17T16:02:04.348295+0000","last_active":"2024-09-17T16:02:44.579626+0000","last_peered":"2024-09-17T16:02:44.579626+0000","last_clean":"2024-09-17T16:02:02.344595+0000","last_became_active":"2024-09-17T16:02:04.348295+0000","last_became_peered":"2024-09-17T16:02:04.348295+0000","last_unstale":"2024-09-17T16:02:44.579626+0000","last_undegraded":"2024-09-17T16:02:04.341600+0000","last_fullsized":"2024-09-17T16:02:04.341131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'31","last_scrub_stamp":"2024-09-17T16:01:58.728617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.728617+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:00.150864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.16505651299999999,"stat_sum":{"num_bytes":5537792,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,8,15,5,9,0],"acting":[2147483647,6,8,15,5,9,0],"avail_no_missing":["6(1)","0(6)","5(4)","8(2)","9(5)","15(3)"],"object_location_counts":[{"shards":"0(6),5(4),6(1),8(2),9(5),15(3)","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"76'18","reported_seq":205,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.534500+0000","last_change":"2024-09-17T16:02:04.349949+0000","last_active":"2024-09-17T16:02:44.534500+0000","last_peered":"2024-09-17T16:02:44.534500+0000","last_clean":"2024-09-17T16:02:02.345085+0000","last_became_active":"2024-09-17T16:02:04.349949+0000","last_became_peered":"2024-09-17T16:02:04.349949+0000","last_unstale":"2024-09-17T16:02:44.534500+0000","last_undegraded":"2024-09-17T16:02:04.342317+0000","last_fullsized":"2024-09-17T16:02:04.342004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.333371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.333371+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.431058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011690380000000001,"stat_sum":{"num_bytes":3538944,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":871,"num_write":4,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,2147483647,1],"acting":[10,4,0,3,5,2147483647,1],"avail_no_missing":["10(0)","0(2)","1(6)","3(3)","4(1)","5(4)"],"object_location_counts":[{"shards":"0(2),1(6),3(3),4(1),5(4),10(0)","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.12","version":"72'39","reported_seq":194,"reported_epoch":106,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:44.542078+0000","last_change":"2024-09-17T16:02:05.063706+0000","last_active":"2024-09-17T16:02:44.542078+0000","last_peered":"2024-09-17T16:02:44.542078+0000","last_clean":"2024-09-17T16:02:02.344908+0000","last_became_active":"2024-09-17T16:02:05.063706+0000","last_became_peered":"2024-09-17T16:02:05.063706+0000","last_unstale":"2024-09-17T16:02:44.542078+0000","last_undegraded":"2024-09-17T16:02:04.342739+0000","last_fullsized":"2024-09-17T16:02:04.342330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'32","last_scrub_stamp":"2024-09-17T16:02:01.034730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.034730+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:06.843079+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0088005819999999995,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,6,12,2,2147483647],"acting":[15,8,5,6,12,2,2147483647],"avail_no_missing":["15(0)","2(5)","5(2)","6(3)","8(1)","12(4)"],"object_location_counts":[{"shards":"2(5),5(2),6(3),8(1),12(4),15(0)","objects":3}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.13","version":"106'46","reported_seq":440,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-17T16:02:44.550012+0000","last_change":"2024-09-17T16:02:44.550883+0000","last_active":"2024-09-17T16:02:44.550012+0000","last_peered":"2024-09-17T16:02:44.550012+0000","last_clean":"2024-09-17T16:02:44.550012+0000","last_became_active":"2024-09-17T16:01:52.366019+0000","last_became_peered":"2024-09-17T16:01:52.366019+0000","last_unstale":"2024-09-17T16:02:44.550012+0000","last_undegraded":"2024-09-17T16:02:44.550012+0000","last_fullsized":"2024-09-17T16:02:44.550012+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'16","last_scrub_stamp":"2024-09-17T16:02:00.060603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.060603+0000","objects_scrubbed":2,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:06.793382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0086514299999999999,"stat_sum":{"num_bytes":917504,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":531,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"1"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":122241024,"num_objects":161,"num_object_clones":111,"num_object_copies":1127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":10,"num_read":290,"num_read_kb":116149,"num_write":541,"num_write_kb":105152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208158720,"data_stored":312397824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1519,"ondisk_log_size":1519,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247830,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41096,"kb_used_data":9908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330744,"statfs":{"total":96636764160,"available":96594681856,"internally_reserved":0,"allocated":10145792,"data_stored":14545558,"data_compressed":261372,"data_compressed_allocated":348160,"data_compressed_original":696320,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280533,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38592,"kb_used_data":11560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333248,"statfs":{"total":96636764160,"available":96597245952,"internally_reserved":0,"allocated":11837440,"data_stored":15196336,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47104,"kb_used_data":15908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324736,"statfs":{"total":96636764160,"available":96588529664,"internally_reserved":0,"allocated":16289792,"data_stored":24312227,"data_compressed":258235,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313238,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48988,"kb_used_data":13636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322852,"statfs":{"total":96636764160,"available":96586600448,"internally_reserved":0,"allocated":13963264,"data_stored":17475501,"data_compressed":251961,"data_compressed_allocated":335872,"data_compressed_original":671744,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280533,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51808,"kb_used_data":20612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320032,"statfs":{"total":96636764160,"available":96583712768,"internally_reserved":0,"allocated":21106688,"data_stored":31154352,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280533,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":19368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":19832832,"data_stored":29470896,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42236,"kb_used_data":11048,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329604,"statfs":{"total":96636764160,"available":96593514496,"internally_reserved":0,"allocated":11313152,"data_stored":17416912,"data_compressed":265595,"data_compressed_allocated":376832,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36272,"kb_used_data":9240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335568,"statfs":{"total":96636764160,"available":96599621632,"internally_reserved":0,"allocated":9461760,"data_stored":15002019,"data_compressed":258235,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":19800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":20275200,"data_stored":31537571,"data_compressed":258235,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46412,"kb_used_data":15220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325428,"statfs":{"total":96636764160,"available":96589238272,"internally_reserved":0,"allocated":15585280,"data_stored":23675056,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48060,"kb_used_data":16868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323780,"statfs":{"total":96636764160,"available":96587550720,"internally_reserved":0,"allocated":17272832,"data_stored":27044259,"data_compressed":258235,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":5900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":6041600,"data_stored":8820876,"data_compressed":114770,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,5,6,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46928,"kb_used_data":15740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324912,"statfs":{"total":96636764160,"available":96588709888,"internally_reserved":0,"allocated":16117760,"data_stored":24850627,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280532,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35988,"kb_used_data":8952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335852,"statfs":{"total":96636764160,"available":96599912448,"internally_reserved":0,"allocated":9166848,"data_stored":10684854,"data_compressed":258235,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37520,"kb_used_data":6324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334320,"statfs":{"total":96636764160,"available":96598343680,"internally_reserved":0,"allocated":6475776,"data_stored":8544432,"data_compressed":255096,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280532,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49548,"kb_used_data":18352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322292,"statfs":{"total":96636764160,"available":96586027008,"internally_reserved":0,"allocated":18792448,"data_stored":28996304,"data_compressed":265595,"data_compressed_allocated":376832,"data_compressed_original":1191936,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19664896,"data_stored":29257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12759040,"data_stored":16568320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11145216,"data_stored":16445440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8232960,"data_stored":14073856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20025344,"data_stored":31236096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15568896,"data_stored":22851584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16044032,"data_stored":26116096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5378048,"data_stored":8372224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14544896,"data_stored":23105536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":7933952,"data_stored":9756672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5259264,"data_stored":7626752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":18624512,"data_stored":28495872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":8904704,"data_stored":13606912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":18259968,"data_stored":27443200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":15060992,"data_stored":23031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":10752000,"data_stored":14409728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:02:48.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.17 3.6 3.7 3.1 3.a 3.f 3.11 2024-09-17T16:02:48.832 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:48.832 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:48.832 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-09-17T16:02:48.832 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:48.832 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:48.830+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:48.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:48.922 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:48.920+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:48.942 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:48.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:48.980 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:48.981 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:49.018 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-09-17T16:02:49.019 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:02:49.022 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:49.020+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:49.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:49.040 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:49.040 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:49.062 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:49.062 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:49.102 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:49.123 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:49.121+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:49.124 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:49.124 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:49.124 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:49.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:49.145 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:49.148 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.148 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:49.167 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:49.168 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:49.182 INFO:teuthology.orchestra.run.smithi006.stderr:instructing pg(s) [3.fs0] on osd.0 to force-backfill; instructing pg(s) [3.11s0] on osd.10 to force-backfill; instructing pg(s) [3.as0] on osd.12 to force-backfill; instructing pg(s) [3.1s0,3.6s0,3.7s0,3.17s0] on osd.13 to force-backfill; 2024-09-17T16:02:49.183 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:02:49.194 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:49.194 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:49.215 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:49.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:49.224 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:49.221+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:49.236 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:49.237 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:49.258 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:49.258 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:49.276 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.276 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.280 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:49.281 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:49.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:49.286 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:49.284+0000 7f93f1d9b640 0 -- 172.21.15.6:0/1010037609 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7f93f4222080 msgr2=0x7f93f42244b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:49.308 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:49.308 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:49.311 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:49.325 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:49.323+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:49.393 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.393 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:49.446 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:49.444+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:49.509 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.509 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:49.559 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:49.557+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:49.565 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:49.572 INFO:tasks.rados.rados.0.smithi181.stdout:373: done (4 left) 2024-09-17T16:02:49.572 INFO:tasks.rados.rados.0.smithi181.stdout:374: done (3 left) 2024-09-17T16:02:49.573 INFO:tasks.rados.rados.0.smithi181.stdout:375: done (2 left) 2024-09-17T16:02:49.573 INFO:tasks.rados.rados.0.smithi181.stdout:376: done (1 left) 2024-09-17T16:02:49.573 INFO:tasks.rados.rados.0.smithi181.stdout:377: done (0 left) 2024-09-17T16:02:49.573 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 44 current snap is 44 2024-09-17T16:02:49.573 INFO:tasks.rados.rados.0.smithi181.stdout:378: seq_num 141 ranges {1736704=425984,2162688=573440,2736128=425984,3162112=524288,3686400=425984} 2024-09-17T16:02:49.583 INFO:tasks.rados.rados.0.smithi181.stdout:378: writing smithi18136759-44 from 1736704 to 2162688 tid 1 2024-09-17T16:02:49.587 INFO:tasks.rados.rados.0.smithi181.stdout:378: writing smithi18136759-44 from 2162688 to 2736128 tid 2 2024-09-17T16:02:49.589 INFO:tasks.rados.rados.0.smithi181.stdout:378: writing smithi18136759-44 from 2736128 to 3162112 tid 3 2024-09-17T16:02:49.592 INFO:tasks.rados.rados.0.smithi181.stdout:378: writing smithi18136759-44 from 3162112 to 3686400 tid 4 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:378: writing smithi18136759-44 from 3686400 to 4112384 tid 5 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:379: rmattr oid 25 current snap is 44 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:379: done (1 left) 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:380: snap_create 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 1 to smithi18136759-44 2024-09-17T16:02:49.595 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 2 to smithi18136759-44 2024-09-17T16:02:49.597 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 3 to smithi18136759-44 2024-09-17T16:02:49.603 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 4 to smithi18136759-44 2024-09-17T16:02:49.603 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 5 to smithi18136759-44 2024-09-17T16:02:49.628 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:49.626+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:49.639 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.639 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:49.729 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:49.727+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:49.758 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.758 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:49.830 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:49.828+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:49.874 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.874 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:49.922 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 6 to smithi18136759-44 2024-09-17T16:02:49.923 INFO:tasks.rados.rados.0.smithi181.stdout:378: finishing write tid 7 to smithi18136759-44 2024-09-17T16:02:49.923 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 updating version 0 to 68 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 updating version 68 to 70 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 updating version 70 to 71 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 updating version 71 to 72 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 version 72 is already newer than 69 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: oid 44 version 72 is already newer than 67 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 44 v 72 (ObjNum 141 snap 44 seq_num 141) dirty exists 2024-09-17T16:02:49.924 INFO:tasks.rados.rados.0.smithi181.stdout:378: left oid 44 (ObjNum 141 snap 44 seq_num 141) 2024-09-17T16:02:49.930 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:49.928+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:49.990 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:49.990 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:49.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:50.031 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:50.029+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:50.106 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.106 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:50.133 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:50.131+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:50.221 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.221 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:50.233 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:50.231+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:50.334 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:50.332+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:50.337 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.337 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:50.434 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:50.431+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:50.453 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.453 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:50.535 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:50.533+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:50.574 INFO:tasks.rados.rados.0.smithi181.stdout:378: done (1 left) 2024-09-17T16:02:50.574 INFO:tasks.rados.rados.0.smithi181.stdout:380: done (0 left) 2024-09-17T16:02:50.574 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 15 current snap is 45 2024-09-17T16:02:50.574 INFO:tasks.rados.rados.0.smithi181.stdout:381: seq_num 142 ranges {245760=442368,688128=720896,1409024=540672,1949696=688128} 2024-09-17T16:02:50.577 INFO:tasks.rados.rados.0.smithi181.stdout:381: writing smithi18136759-15 from 245760 to 688128 tid 1 2024-09-17T16:02:50.582 INFO:tasks.rados.rados.0.smithi181.stdout:381: writing smithi18136759-15 from 688128 to 1409024 tid 2 2024-09-17T16:02:50.585 INFO:tasks.rados.rados.0.smithi181.stdout:381: writing smithi18136759-15 from 1409024 to 1949696 tid 3 2024-09-17T16:02:50.589 INFO:tasks.rados.rados.0.smithi181.stdout:381: writing smithi18136759-15 from 1949696 to 2637824 tid 4 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:382: rmattr oid 7 current snap is 45 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:382: done (1 left) 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:383: copy_from oid 8 from oid 9 current snap is 45 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 1 to smithi18136759-15 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 2 to smithi18136759-15 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 3 to smithi18136759-15 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:384: read oid 16 snap 35 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:384: expect (ObjNum 1474905552 snap 21989 seq_num 1474947680) 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:385: copy_from oid 6 from oid 21 current snap is 45 2024-09-17T16:02:50.590 INFO:tasks.rados.rados.0.smithi181.stdout:386: snap_create 2024-09-17T16:02:50.591 INFO:tasks.rados.rados.0.smithi181.stdout:384: expect (ObjNum 1474905552 snap 21989 seq_num 1474947680) 2024-09-17T16:02:50.597 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 4 to smithi18136759-15 2024-09-17T16:02:50.599 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 5 to smithi18136759-15 2024-09-17T16:02:50.600 INFO:tasks.rados.rados.0.smithi181.stdout:381: finishing write tid 6 to smithi18136759-15 2024-09-17T16:02:50.600 INFO:tasks.rados.rados.0.smithi181.stdout:381: oid 15 updating version 0 to 53 2024-09-17T16:02:50.600 INFO:tasks.rados.rados.0.smithi181.stdout:381: oid 15 version 53 is already newer than 49 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:381: oid 15 version 53 is already newer than 51 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:381: oid 15 version 53 is already newer than 52 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:381: oid 15 version 53 is already newer than 50 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 15 v 53 (ObjNum 142 snap 45 seq_num 142) dirty exists 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:381: left oid 15 (ObjNum 142 snap 45 seq_num 142) 2024-09-17T16:02:50.601 INFO:tasks.rados.rados.0.smithi181.stdout:385: finishing copy_from racing read to smithi18136759-6 2024-09-17T16:02:50.605 INFO:tasks.rados.rados.0.smithi181.stdout:385: finishing copy_from to smithi18136759-6 2024-09-17T16:02:50.605 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 39 (ObjNum 65 snap 0 seq_num 1818521955) dirty exists 2024-09-17T16:02:50.608 INFO:tasks.rados.rados.0.smithi181.stdout:383: finishing copy_from racing read to smithi18136759-8 2024-09-17T16:02:50.614 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.615 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.619 INFO:tasks.rados.rados.0.smithi181.stdout:383: finishing copy_from to smithi18136759-8 2024-09-17T16:02:50.619 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 56 (ObjNum 118 snap 29 seq_num 118) dirty exists 2024-09-17T16:02:50.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:50.668 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:50.665+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:50.730 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.730 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.737 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:50.735+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:50.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:50.837 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:50.836+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:50.886 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:50.886 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:50.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:50.938 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:50.936+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:51.001 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:51.002 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:51.010 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:51.055 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:51.053+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:51.125 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:51.126 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:51.134 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:51.140 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:51.138+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:51.241 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:51.239+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:51.278 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:51.279 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:51.287 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:51.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-17T16:02:51.341 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:51.339+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:51.391 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:51.391 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:51.400 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:51.444 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:51.442+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:51.459 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:51.459 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:51.482 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:51.482 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:51.504 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:51.505 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:51.507 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:51.507 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:51.516 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:51.526 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:51.526 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:51.547 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:51.557 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:51.555+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:51.568 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:51.568 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:51.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:381: done (4 left) 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:383: done (3 left) 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:384: done (2 left) 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:385: done (1 left) 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:386: done (0 left) 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:387: copy_from oid 47 from oid 36 current snap is 46 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 9 current snap is 46 2024-09-17T16:02:51.583 INFO:tasks.rados.rados.0.smithi181.stdout:388: seq_num 143 ranges {1998848=458752,2457600=475136,2932736=655360,3588096=671744,4259840=147456} 2024-09-17T16:02:51.596 INFO:tasks.rados.rados.0.smithi181.stdout:388: writing smithi18136759-9 from 1998848 to 2457600 tid 1 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:51.598 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:51.598 INFO:tasks.rados.rados.0.smithi181.stdout:388: writing smithi18136759-9 from 2457600 to 2932736 tid 2 2024-09-17T16:02:51.602 INFO:tasks.rados.rados.0.smithi181.stdout:388: writing smithi18136759-9 from 2932736 to 3588096 tid 3 2024-09-17T16:02:51.606 INFO:tasks.rados.rados.0.smithi181.stdout:388: writing smithi18136759-9 from 3588096 to 4259840 tid 4 2024-09-17T16:02:51.607 INFO:tasks.rados.rados.0.smithi181.stdout:388: writing smithi18136759-9 from 4259840 to 4407296 tid 5 2024-09-17T16:02:51.607 INFO:tasks.rados.rados.0.smithi181.stdout:389: copy_from oid 34 from oid 35 current snap is 46 2024-09-17T16:02:51.607 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 1 to smithi18136759-9 2024-09-17T16:02:51.607 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 2 to smithi18136759-9 2024-09-17T16:02:51.607 INFO:tasks.rados.rados.0.smithi181.stdout:390: setattr oid 1 current snap is 46 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:391: read oid 46 snap -1 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:391: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:append oid 28 current snap is 46 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:392: seq_num 144 ranges {0=16384} 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:392: writing smithi18136759-28 from 0 to 16384 tid 1 2024-09-17T16:02:51.609 INFO:tasks.rados.rados.0.smithi181.stdout:393: setattr oid 16 current snap is 46 2024-09-17T16:02:51.610 INFO:tasks.rados.rados.0.smithi181.stdout:394: copy_from oid 13 from oid 25 current snap is 46 2024-09-17T16:02:51.610 INFO:tasks.rados.rados.0.smithi181.stdout:395: rollback oid 6 current snap is 46 2024-09-17T16:02:51.611 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 6 to 44 2024-09-17T16:02:51.611 INFO:tasks.rados.rados.0.smithi181.stdout:396: rollback oid 19 current snap is 46 2024-09-17T16:02:51.611 INFO:tasks.rados.rados.0.smithi181.stdout:rollback oid 19 to 40 2024-09-17T16:02:51.611 INFO:tasks.rados.rados.0.smithi181.stdout:397: snap_create 2024-09-17T16:02:51.612 INFO:tasks.rados.rados.0.smithi181.stdout:387: finishing copy_from racing read to smithi18136759-47 2024-09-17T16:02:51.614 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 3 to smithi18136759-9 2024-09-17T16:02:51.617 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 1 v 70 (ObjNum 108 snap 26 seq_num 108) dirty exists 2024-09-17T16:02:51.618 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 4 to smithi18136759-9 2024-09-17T16:02:51.618 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 5 to smithi18136759-9 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 6 to smithi18136759-9 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:392: finishing write tid 1 to smithi18136759-28 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: finishing write tid 7 to smithi18136759-9 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 updating version 0 to 69 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 version 69 is already newer than 65 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 version 69 is already newer than 68 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 version 69 is already newer than 64 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 version 69 is already newer than 66 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: oid 9 version 69 is already newer than 67 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 9 v 69 (ObjNum 143 snap 46 seq_num 143) dirty exists 2024-09-17T16:02:51.620 INFO:tasks.rados.rados.0.smithi181.stdout:388: left oid 9 (ObjNum 143 snap 46 seq_num 143) 2024-09-17T16:02:51.621 INFO:tasks.rados.rados.0.smithi181.stdout:387: finishing copy_from to smithi18136759-47 2024-09-17T16:02:51.621 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 47 v 143 (ObjNum 133 snap 40 seq_num 133) dirty exists 2024-09-17T16:02:51.622 INFO:tasks.rados.rados.0.smithi181.stdout:396: finishing rollback tid 0 to smithi18136759-19 2024-09-17T16:02:51.622 INFO:tasks.rados.rados.0.smithi181.stdout:396: finishing rollback tid 1 to smithi18136759-19 2024-09-17T16:02:51.622 INFO:tasks.rados.rados.0.smithi181.stdout:392: finishing write tid 2 to smithi18136759-28 2024-09-17T16:02:51.622 INFO:tasks.rados.rados.0.smithi181.stdout:395: finishing rollback tid 0 to smithi18136759-6 2024-09-17T16:02:51.623 INFO:tasks.rados.rados.0.smithi181.stdout:396: finishing rollback tid 2 to smithi18136759-19 2024-09-17T16:02:51.623 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 19 v 175 (ObjNum 136 snap 40 seq_num 136) dirty exists 2024-09-17T16:02:51.623 INFO:tasks.rados.rados.0.smithi181.stdout:389: finishing copy_from racing read to smithi18136759-34 2024-09-17T16:02:51.623 INFO:tasks.rados.rados.0.smithi181.stdout:395: finishing rollback tid 1 to smithi18136759-6 2024-09-17T16:02:51.624 INFO:tasks.rados.rados.0.smithi181.stdout:389: finishing copy_from to smithi18136759-34 2024-09-17T16:02:51.624 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 34 v 23 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:392: finishing write tid 3 to smithi18136759-28 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:392: oid 28 updating version 0 to 56 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:392: oid 28 version 56 is already newer than 55 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 28 v 56 (ObjNum 144 snap 46 seq_num 144) dirty exists 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:392: left oid 28 (ObjNum 144 snap 46 seq_num 144) 2024-09-17T16:02:51.625 INFO:tasks.rados.rados.0.smithi181.stdout:394: finishing copy_from racing read to smithi18136759-13 2024-09-17T16:02:51.626 INFO:tasks.rados.rados.0.smithi181.stdout:394: finishing copy_from to smithi18136759-13 2024-09-17T16:02:51.626 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 13 v 4 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T16:02:51.631 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:51.631 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:51.633 INFO:tasks.rados.rados.0.smithi181.stdout:391: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-17T16:02:51.633 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:51.640 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:51.643 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:51.641+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:51.647 INFO:tasks.rados.rados.0.smithi181.stdout:seeking to 0 2024-09-17T16:02:51.652 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:51.672 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:51.672 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:51.673 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:51.673 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:51.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:51.689 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:51.689 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:51.714 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:51.714 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:51.735 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:51.735 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:51.743 INFO:tasks.ceph.osd.6.smithi169.stderr:2024-09-17T16:02:51.742+0000 7fd1dafee640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33025) UID: 0 2024-09-17T16:02:51.756 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:51.757 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:51.777 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:2024-09-17T16:02:51.776+0000 7fa2c954f640 0 -- 172.21.15.6:0/1908611109 >> [v2:172.21.15.181:6802/2815528414,v1:172.21.15.181:6803/2815528414] conn(0x7fa2c421d620 msgr2=0x7fa2c421fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 46/32 ECSubWrite(tid=38, reqid=osd.0.0:38, at_version=46'16, trim_to=0'0, roll_forward_to=44'12))", 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 66.859246691999999, 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071159159999999999, 2024-09-17T16:02:51.787 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925029+0000", 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8899999999999999e-07 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925052+0000", 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2929999999999999e-05 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925081+0000", 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8979000000000002e-05 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925089+0000", 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3359999999999995e-06 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925144+0000", 2024-09-17T16:02:51.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4988999999999998e-05 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.925153+0000", 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6289999999999999e-06 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932110+0000", 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069573769999999998 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.932145+0000", 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4087000000000001e-05 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.as5 46/32 ECSubWrite(tid=53, reqid=osd.12.0:53, at_version=46'16, trim_to=0'0, roll_forward_to=43'12))", 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 66.856887729999997, 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0076133829999999996, 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.792 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927388+0000", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6499999999999998e-07 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927409+0000", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0454000000000001e-05 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927438+0000", 2024-09-17T16:02:51.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8751999999999998e-05 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927444+0000", 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6409999999999996e-06 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927496+0000", 2024-09-17T16:02:51.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1591000000000001e-05 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.927505+0000", 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6290000000000005e-06 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.934970+0000", 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0074642270000000004 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.935001+0000", 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1324e-05 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 64/53 ECSubWrite(tid=26, reqid=osd.8.0:26, at_version=64'12, trim_to=0'0, roll_forward_to=62'10))", 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 49.438509177999997, 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087141639999999999, 2024-09-17T16:02:51.797 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345766+0000", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5600000000000001e-07 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.345787+0000", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0109e-05 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346312+0000", 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00052588999999999999 2024-09-17T16:02:51.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346324+0000", 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1445e-05 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346394+0000", 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9604000000000006e-05 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346403+0000", 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.611e-06 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354448+0000", 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080448199999999994 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354480+0000", 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2329000000000002e-05 2024-09-17T16:02:51.801 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 64/32 ECSubWrite(tid=69, reqid=osd.0.0:69, at_version=64'28, trim_to=0'0, roll_forward_to=62'26))", 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 49.438076840000001, 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082305039999999996, 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.802 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346198+0000", 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346199+0000", 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7300000000000002e-07 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346223+0000", 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4442000000000001e-05 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346360+0000", 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000136792 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346374+0000", 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3669000000000001e-05 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346429+0000", 2024-09-17T16:02:51.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4898999999999997e-05 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346437+0000", 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3520000000000007e-06 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354393+0000", 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079559079999999994 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354429+0000", 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6168999999999999e-05 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.10s1 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:51.807 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 49.438037839000003, 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083156649999999999, 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346237+0000", 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346238+0000", 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8300000000000001e-07 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346256+0000", 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8805999999999999e-05 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346408+0000", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000151519 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346417+0000", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.6619999999999994e-06 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346513+0000", 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6127999999999998e-05 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.346523+0000", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9750000000000002e-06 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354521+0000", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079977780000000005 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.354553+0000", 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2614000000000003e-05 2024-09-17T16:02:51.811 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.18s6 81/53 ECSubWrite(tid=277, reqid=osd.11.0:277, at_version=81'62, trim_to=0'0, roll_forward_to=79'60))", 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.35302557, 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010651089000000001, 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431250+0000", 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4499999999999998e-07 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431286+0000", 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6235000000000001e-05 2024-09-17T16:02:51.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431365+0000", 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8455999999999996e-05 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431373+0000", 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5860000000000005e-06 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431445+0000", 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1258000000000001e-05 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431455+0000", 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.007e-05 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441861+0000", 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010406072000000001 2024-09-17T16:02:51.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441901+0000", 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0166999999999997e-05 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.8s6 81/53 ECSubWrite(tid=278, reqid=osd.11.0:278, at_version=81'50, trim_to=0'0, roll_forward_to=77'48))", 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.352983887999997, 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010640938000000001, 2024-09-17T16:02:51.817 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431291+0000", 2024-09-17T16:02:51.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431292+0000", 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0199999999999998e-07 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431315+0000", 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2889000000000001e-05 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431418+0000", 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103299 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.820 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431425+0000", 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9430000000000003e-06 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431465+0000", 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0303999999999999e-05 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.431473+0000", 2024-09-17T16:02:51.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2770000000000005e-06 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441909+0000", 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.01043585 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.441932+0000", 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3074e-05 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.823 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 81/32 ECSubWrite(tid=100, reqid=osd.0.0:100, at_version=81'47, trim_to=0'0, roll_forward_to=79'45))", 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.350400510999997, 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010862208999999999, 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433875+0000", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.48e-07 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433889+0000", 2024-09-17T16:02:51.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3553000000000001e-05 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433919+0000", 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0408000000000001e-05 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433925+0000", 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5890000000000002e-06 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433972+0000", 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7898000000000001e-05 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.433983+0000", 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0239e-05 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444701+0000", 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010717925999999999 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444737+0000", 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6448000000000001e-05 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs4 81/32 ECSubWrite(tid=101, reqid=osd.0.0:101, at_version=81'49, trim_to=0'0, roll_forward_to=79'45))", 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.348399518999997, 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0098407489999999993, 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.828 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.829 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435876+0000", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9599999999999999e-07 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435897+0000", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0579999999999999e-05 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435930+0000", 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2873000000000002e-05 2024-09-17T16:02:51.830 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435938+0000", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5709999999999998e-06 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435990+0000", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1897000000000001e-05 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.435998+0000", 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6520000000000006e-06 2024-09-17T16:02:51.831 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445684+0000", 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0096856419999999995 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445717+0000", 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2938000000000002e-05 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.832 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 81/53 ECSubWrite(tid=45, reqid=osd.8.0:45, at_version=81'22, trim_to=0'0, roll_forward_to=77'20))", 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.347516560999999, 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0090338850000000002, 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.833 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436759+0000", 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1099999999999995e-07 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.834 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436775+0000", 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5614e-05 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436802+0000", 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6704999999999999e-05 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436818+0000", 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5858e-05 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.835 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436875+0000", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7402999999999999e-05 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.436885+0000", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7860000000000008e-06 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445759+0000", 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0088739249999999995 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.836 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445793+0000", 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3983000000000003e-05 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 81/53 ECSubWrite(tid=46, reqid=osd.8.0:46, at_version=81'22, trim_to=0'0, roll_forward_to=77'20))", 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.347229048000003, 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0087742510000000003, 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.837 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437046+0000", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.74e-07 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437060+0000", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3212e-05 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437074+0000", 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4769999999999999e-05 2024-09-17T16:02:51.839 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437082+0000", 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.4540000000000001e-06 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437128+0000", 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6455000000000002e-05 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.840 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437136+0000", 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1740000000000003e-06 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445796+0000", 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086608529999999996 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445821+0000", 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4159e-05 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.841 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=101, reqid=osd.2.0:100, at_version=0'0, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.346772545999997, 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078549550000000003, 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.842 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437503+0000", 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3499999999999996e-07 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.843 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437511+0000", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8709999999999997e-06 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437539+0000", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.7525000000000002e-05 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437545+0000", 2024-09-17T16:02:51.844 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8100000000000003e-06 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437582+0000", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7070000000000003e-05 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.437587+0000", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3070000000000002e-06 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445327+0000", 2024-09-17T16:02:51.845 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077395270000000004 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445358+0000", 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1210000000000001e-05 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=102, reqid=osd.2.0:102, at_version=81'87, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.345616213, 2024-09-17T16:02:51.846 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0080538369999999995, 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438659+0000", 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.847 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438660+0000", 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.540000000000001e-07 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438689+0000", 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8722e-05 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.848 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438734+0000", 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5291999999999999e-05 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438742+0000", 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1729999999999995e-06 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438789+0000", 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6764999999999997e-05 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.849 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438798+0000", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3079999999999997e-06 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446667+0000", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0078683329999999999 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446713+0000", 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6289999999999999e-05 2024-09-17T16:02:51.850 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 81/32 ECSubWrite(tid=103, reqid=osd.2.0:103, at_version=81'89, trim_to=0'0, roll_forward_to=81'85))", 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.344570517999998, 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0070349870000000004, 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.851 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439705+0000", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439706+0000", 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8999999999999995e-07 2024-09-17T16:02:51.852 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439725+0000", 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8916999999999999e-05 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439760+0000", 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5351000000000001e-05 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.853 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439782+0000", 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.1793e-05 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442292+0000", 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025106099999999999 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.442302+0000", 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8660000000000001e-06 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.854 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446717+0000", 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0044147750000000001 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.446740+0000", 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2884999999999999e-05 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.es5 97/32 ECSubWrite(tid=136, reqid=osd.2.0:136, at_version=97'121, trim_to=0'0, roll_forward_to=93'117))", 2024-09-17T16:02:51.855 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:35.510973+0000", 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.273302191999999, 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006461044, 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.510973+0000", 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.856 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.510973+0000", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.510974+0000", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.074e-06 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.511028+0000", 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3797e-05 2024-09-17T16:02:51.857 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.511327+0000", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00029852399999999999 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.511337+0000", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0849000000000001e-05 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.511404+0000", 2024-09-17T16:02:51.858 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6903999999999995e-05 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.511586+0000", 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018174 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517394+0000", 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058080880000000003 2024-09-17T16:02:51.859 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517434+0000", 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0068000000000003e-05 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.5s6 97/53 ECSubWrite(tid=66, reqid=osd.8.0:66, at_version=97'30, trim_to=0'0, roll_forward_to=93'28))", 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:35.516996+0000", 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 16.267279340999998, 2024-09-17T16:02:51.860 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063366840000000004, 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.516996+0000", 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.516996+0000", 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.861 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.516996+0000", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2499999999999999e-07 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517022+0000", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5714000000000001e-05 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517059+0000", 2024-09-17T16:02:51.862 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7403000000000001e-05 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517067+0000", 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2289999999999997e-06 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517150+0000", 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3540000000000003e-05 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.863 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.517163+0000", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2638e-05 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.523286+0000", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061236939999999998 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:35.523333+0000", 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6241e-05 2024-09-17T16:02:51.864 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 98/53 ECSubWrite(tid=68, reqid=osd.8.0:68, at_version=98'34, trim_to=0'0, roll_forward_to=93'32))", 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:36.511546+0000", 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 15.272729787999999, 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065419379999999997, 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.865 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511546+0000", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511546+0000", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511546+0000", 2024-09-17T16:02:51.866 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9999999999999998e-07 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511581+0000", 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5101000000000002e-05 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511638+0000", 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7045e-05 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.867 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.511650+0000", 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2114e-05 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.512236+0000", 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000585467 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.512249+0000", 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3505e-05 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.868 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.518038+0000", 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057890679999999996 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:36.518087+0000", 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9237999999999999e-05 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.869 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:866 3.10s1 3:0a7b2816:::smithi18136759-31:head [copy-from ver 2] snapc 29=[28,26,25,22,21,1c,19,18,16,c,6] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:42.535315+0000", 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.2489601849999996, 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.015324931, 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 866 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.870 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535315+0000", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535315+0000", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535317+0000", 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8199999999999999e-06 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.871 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535325+0000", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1540000000000002e-06 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535343+0000", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7761000000000002e-05 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535352+0000", 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4110000000000002e-06 2024-09-17T16:02:51.872 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535445+0000", 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.2487999999999999e-05 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535567+0000", 2024-09-17T16:02:51.873 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012244900000000001 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.547984+0000", 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012417052 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.549288+0000", 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013040059999999999 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:51.874 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.550519+0000", 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012309689999999999 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.550640+0000", 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120821 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:867 3.10s1 3:0a7b2816:::smithi18136759-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e104)", 2024-09-17T16:02:51.875 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:42.535378+0000", 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 9.2488976189999992, 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.013867343000000001, 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 867 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.876 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535378+0000", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535378+0000", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535378+0000", 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0100000000000004e-07 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.877 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535382+0000", 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1719999999999999e-06 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535392+0000", 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0317e-05 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.878 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535400+0000", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8290000000000008e-06 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535889+0000", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048914100000000001 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for blocked object", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.535947+0000", 2024-09-17T16:02:51.879 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8508000000000003e-05 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.548869+0000", 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012921861999999999 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.548992+0000", 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000122478 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.880 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:42.549245+0000", 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025333499999999999 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.15s6 113/53 ECSubWrite(tid=92, reqid=client.4560.0:949, at_version=113'42, trim_to=0'0, roll_forward_to=107'40))", 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:51.608019+0000", 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 0.17625663499999999, 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.012030152000000001, 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:51.881 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.608019+0000", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.608019+0000", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:51.882 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.608030+0000", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1039000000000001e-05 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.611987+0000", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0039572829999999998 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.612019+0000", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1854999999999999e-05 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:51.883 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.612027+0000", 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5059999999999995e-06 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.613058+0000", 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010308870000000001 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.613070+0000", 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1426999999999999e-05 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.884 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.619972+0000", 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069027070000000001 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:51.620049+0000", 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6447999999999997e-05 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:51.885 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:51.887 INFO:tasks.rados.rados.0.smithi181.stdout:395: finishing rollback tid 2 to smithi18136759-6 2024-09-17T16:02:51.887 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 6 v 73 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T16:02:51.887 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 16 v 41 (ObjNum 65 snap 0 seq_num 1454765552) dirty exists 2024-09-17T16:02:51.887 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:51.888 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:51.885+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:51.988 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:51.986+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:52.002 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:52.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:52.033 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:52.033 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:52.033 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.042 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:52.089 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:52.087+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:52.184 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.185 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:52.185 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:52.185 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:52.185 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.189 INFO:tasks.ceph.osd.4.smithi006.stderr:2024-09-17T16:02:52.187+0000 7f8d29da7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33620) UID: 0 2024-09-17T16:02:52.193 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:52.290 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:52.288+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:52.296 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.296 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:52.296 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:52.296 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:52.296 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=1, reqid=client.4560.0:19, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.438790591, 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0077409619999999997, 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.297 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.854983+0000", 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.298 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855037+0000", 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4381e-05 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855095+0000", 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8153999999999999e-05 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855108+0000", 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2683e-05 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.299 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855262+0000", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00015417100000000001 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855288+0000", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5622e-05 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862675+0000", 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0073876100000000002 2024-09-17T16:02:52.300 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862723+0000", 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8100999999999999e-05 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 34/32 ECSubWrite(tid=3, reqid=client.4560.0:20, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.437889717000004, 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068711939999999997, 2024-09-17T16:02:52.301 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855883+0000", 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.302 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855885+0000", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1930000000000001e-06 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855911+0000", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6440000000000001e-05 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855960+0000", 2024-09-17T16:02:52.303 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.9356e-05 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855971+0000", 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0722e-05 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856089+0000", 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000117887 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.304 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856101+0000", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1891e-05 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862730+0000", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0066287849999999999 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862755+0000", 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.4919999999999999e-05 2024-09-17T16:02:52.305 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.9s6 35/32 ECSubWrite(tid=4, reqid=client.4560.0:37, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.433799890000003, 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068557749999999997, 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.306 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859973+0000", 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.307 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859974+0000", 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.96e-07 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860071+0000", 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.7822e-05 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.308 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860136+0000", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4172999999999997e-05 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860152+0000", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6211000000000001e-05 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860251+0000", 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8916e-05 2024-09-17T16:02:52.309 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.860257+0000", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6159999999999998e-06 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866786+0000", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.006528832 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.866829+0000", 2024-09-17T16:02:52.310 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2809000000000003e-05 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=2, reqid=client.4560.0:17, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.431392928999998, 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089559010000000005, 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.311 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862380+0000", 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.312 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862381+0000", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7999999999999996e-07 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862395+0000", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3890000000000001e-05 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862878+0000", 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00048315400000000002 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.313 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862890+0000", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1965e-05 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862956+0000", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5922999999999999e-05 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862965+0000", 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4089999999999992e-06 2024-09-17T16:02:52.314 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871306+0000", 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0083410649999999999 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871336+0000", 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0015e-05 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.315 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=4, reqid=client.4560.0:25, at_version=34'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.429567551000005, 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0071503649999999997, 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.316 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864206+0000", 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7199999999999998e-07 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.317 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864290+0000", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3232999999999994e-05 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864320+0000", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0243000000000002e-05 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864326+0000", 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6449999999999999e-06 2024-09-17T16:02:52.318 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864394+0000", 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7623e-05 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864405+0000", 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1218e-05 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.319 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871339+0000", 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069335450000000002 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871356+0000", 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7085999999999999e-05 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.fs5 34/32 ECSubWrite(tid=3, reqid=client.4560.0:32, at_version=34'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.320 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.429419476000007, 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0069336040000000003, 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.321 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864354+0000", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9699999999999996e-07 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864368+0000", 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.419e-05 2024-09-17T16:02:52.322 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864398+0000", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9696000000000002e-05 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864407+0000", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1759999999999999e-06 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864473+0000", 2024-09-17T16:02:52.323 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5456999999999993e-05 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864479+0000", 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1240000000000002e-06 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871244+0000", 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0067650319999999998 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.324 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871287+0000", 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.3331999999999998e-05 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.cs3 34/32 ECSubWrite(tid=5, reqid=client.4560.0:26, at_version=34'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.428769591999995, 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063969220000000002, 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.325 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.326 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865004+0000", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2399999999999997e-07 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865031+0000", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6308e-05 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865063+0000", 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2882999999999997e-05 2024-09-17T16:02:52.327 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865072+0000", 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7600000000000008e-06 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865138+0000", 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5407000000000006e-05 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.328 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.865151+0000", 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3115000000000001e-05 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871359+0000", 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0062081330000000002 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871400+0000", 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1591999999999998e-05 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.329 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.426192228999994, 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060965799999999999, 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.330 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867581+0000", 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0600000000000001e-07 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.331 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867659+0000", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8248e-05 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867741+0000", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.1703e-05 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867749+0000", 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.9990000000000008e-06 2024-09-17T16:02:52.332 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867826+0000", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7214999999999998e-05 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.867836+0000", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5489999999999995e-06 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873640+0000", 2024-09-17T16:02:52.333 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058039559999999999 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873677+0000", 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7604000000000001e-05 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:52.334 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.425701056999998, 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0056322610000000004, 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868072+0000", 2024-09-17T16:02:52.335 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868073+0000", 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2399999999999998e-07 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868090+0000", 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7689999999999998e-05 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.336 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868126+0000", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5150000000000001e-05 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.868135+0000", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871137+0000", 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003002167 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.337 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.871147+0000", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6520000000000001e-06 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873682+0000", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0025353490000000001 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.873704+0000", 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2436e-05 2024-09-17T16:02:52.338 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.6s6 52/32 ECSubWrite(tid=89, reqid=client.4560.0:340, at_version=52'11, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 61.920779647000003, 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15958299200000001, 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.339 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372993+0000", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.372994+0000", 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4600000000000003e-07 2024-09-17T16:02:52.340 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.373011+0000", 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.6280999999999999e-05 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531935+0000", 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.15892432300000001 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.341 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.531949+0000", 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4436e-05 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532026+0000", 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6526000000000006e-05 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532033+0000", 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5789999999999998e-06 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.342 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532538+0000", 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00050494099999999996 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:50.532576+0000", 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8059999999999998e-05 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.343 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.1s4 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 49.955255176999998, 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0058173629999999999, 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.344 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338518+0000", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5199999999999997e-07 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338580+0000", 2024-09-17T16:02:52.345 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1075999999999996e-05 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338633+0000", 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3263999999999997e-05 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.338646+0000", 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3326000000000001e-05 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.346 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339218+0000", 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00057138000000000002 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.339229+0000", 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.1487e-05 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344292+0000", 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050634460000000001 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.347 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:02.344335+0000", 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2831999999999999e-05 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:602 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~606208] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.862155487999999, 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0085762449999999997, 2024-09-17T16:02:52.348 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 602 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.349 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431618+0000", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.431620+0000", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.5469999999999999e-06 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432327+0000", 2024-09-17T16:02:52.350 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000707079 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432340+0000", 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3021e-05 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432358+0000", 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.8019000000000001e-05 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.351 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432480+0000", 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121995 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "op must wait for map", 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432491+0000", 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0457e-05 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434624+0000", 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021328250000000001 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.352 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.434764+0000", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014050999999999999 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.436049+0000", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012844969999999999 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439929+0000", 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038802730000000001 2024-09-17T16:02:52.353 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440073+0000", 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00014401099999999999 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440194+0000", 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000121011 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.354 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:603 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~688128] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.861345178999997, 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0089592730000000002, 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 603 2024-09-17T16:02:52.355 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432428+0000", 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.356 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432429+0000", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432988+0000", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00055916400000000002 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.432997+0000", 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.8310000000000007e-06 2024-09-17T16:02:52.357 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433001+0000", 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6299999999999997e-06 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433056+0000", 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4994999999999998e-05 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:52.358 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433061+0000", 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8040000000000001e-06 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437100+0000", 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0040393169999999997 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437171+0000", 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0548000000000003e-05 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.359 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.437907+0000", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073646200000000001 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439930+0000", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002022565 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441283+0000", 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0013528189999999999 2024-09-17T16:02:52.360 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441387+0000", 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010435 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:604 3.11s0 3:89c7af4b:::smithi18136759-34:head [append 0~475136] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.860757857999999, 2024-09-17T16:02:52.361 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086384329999999992, 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 604 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:52.362 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433015+0000", 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433016+0000", 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6899999999999997e-07 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.363 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433437+0000", 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042084799999999998 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433445+0000", 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.354e-06 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433450+0000", 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1460000000000004e-06 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.364 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433526+0000", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5370000000000005e-05 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.22e-06 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438550+0000", 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0050189079999999999 2024-09-17T16:02:52.365 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.438619+0000", 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9138000000000001e-05 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.439350+0000", 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073165800000000005 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.366 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441219+0000", 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001868747 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441548+0000", 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00032912700000000002 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441654+0000", 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010534799999999999 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.367 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:605 3.11s0 3:89c7af4b:::smithi18136759-34:head [setxattr _header (57) in=7b] snapc 1b=[1b,1a,19,18,16,10,e,d,c,9,6] ondisk+write+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.860312077000003, 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0086379049999999995, 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.368 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 605 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.369 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433461+0000", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433484+0000", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.2540000000000001e-05 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433488+0000", 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.1790000000000001e-06 2024-09-17T16:02:52.370 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433491+0000", 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.915e-06 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433560+0000", 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9307000000000006e-05 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:52.371 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433565+0000", 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8049999999999997e-06 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440435+0000", 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0068700439999999996 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440548+0000", 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000112904 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.372 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.440860+0000", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00031159299999999998 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441710+0000", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00085002099999999996 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441963+0000", 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00025321099999999999 2024-09-17T16:02:52.373 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442099+0000", 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013592399999999999 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:606 3.11s0 3:89c7af4b:::smithi18136759-34:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e76)", 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 37.860254001000001, 2024-09-17T16:02:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.010718325000000001, 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 606 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433519+0000", 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6800000000000001e-07 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.376 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433522+0000", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.052e-06 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433531+0000", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9580000000000002e-06 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433536+0000", 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4750000000000004e-06 2024-09-17T16:02:52.377 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433588+0000", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.2129999999999997e-05 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.433592+0000", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9090000000000002e-06 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441080+0000", 2024-09-17T16:02:52.378 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.007488476 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.441139+0000", 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.8468000000000003e-05 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442343+0000", 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0012042520000000001 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.379 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.442403+0000", 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9846999999999998e-05 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:14.444237+0000", 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00183439 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.380 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=148, reqid=osd.14.0:148, at_version=81'37, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.855632489999998, 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0065669389999999999, 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.381 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438141+0000", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9900000000000002e-07 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438158+0000", 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7248000000000002e-05 2024-09-17T16:02:52.382 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438229+0000", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.1317999999999997e-05 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438248+0000", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.827e-05 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438320+0000", 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2342999999999995e-05 2024-09-17T16:02:52.383 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.438331+0000", 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0969e-05 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444669+0000", 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0063379129999999997 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.384 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.444708+0000", 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8578999999999998e-05 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=150, reqid=osd.14.0:150, at_version=81'39, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.854414396000003, 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0057892539999999998, 2024-09-17T16:02:52.385 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.386 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439359+0000", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6700000000000003e-07 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439380+0000", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.0604000000000001e-05 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439418+0000", 2024-09-17T16:02:52.387 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7707000000000003e-05 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439426+0000", 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3289999999999994e-06 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439492+0000", 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5801000000000002e-05 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.388 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.439505+0000", 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2699e-05 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445102+0000", 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055975479999999999 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.389 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445148+0000", 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.5899000000000002e-05 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=152, reqid=client.4560.0:670, at_version=81'41, trim_to=0'0, roll_forward_to=77'35))", 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.848169161999998, 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082580629999999995, 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.390 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445604+0000", 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.391 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445605+0000", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.029e-06 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445725+0000", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000120361 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445763+0000", 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.7991000000000003e-05 2024-09-17T16:02:52.392 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445773+0000", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0039e-05 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445830+0000", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6909e-05 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.445841+0000", 2024-09-17T16:02:52.393 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0327e-05 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453823+0000", 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0079825799999999995 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.453862+0000", 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8826999999999999e-05 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.394 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.19s6 81/53 ECSubWrite(tid=153, reqid=client.4560.0:671, at_version=81'42, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 32.845132442000001, 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0055985469999999997, 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:52.395 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448641+0000", 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6700000000000003e-07 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.396 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448735+0000", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3511000000000006e-05 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448773+0000", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8169e-05 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448782+0000", 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.7509999999999997e-06 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.397 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448846+0000", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.4108000000000004e-05 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.448877+0000", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1097999999999999e-05 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454215+0000", 2024-09-17T16:02:52.398 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0053383129999999999 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:19.454239+0000", 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3830000000000001e-05 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.399 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.401 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:52.402 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:52.399+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:52.502 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:52.500+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:52.504 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:52.504 INFO:teuthology.orchestra.run.smithi169.stdout: "num_ops": 0 2024-09-17T16:02:52.504 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:52.582 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:387: done (10 left) 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:388: done (9 left) 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:389: done (8 left) 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:390: done (7 left) 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:391: done (6 left) 2024-09-17T16:02:52.586 INFO:tasks.rados.rados.0.smithi181.stdout:392: done (5 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:393: done (4 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:394: done (3 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:395: done (2 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:396: done (1 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:397: done (0 left) 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:398: copy_from oid 8 from oid 32 current snap is 47 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout:399: rmattr oid 27 current snap is 47 2024-09-17T16:02:52.587 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 2 2024-09-17T16:02:52.592 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 27 v 41 (ObjNum 100 snap 24 seq_num 100) dirty exists 2024-09-17T16:02:52.593 INFO:tasks.rados.rados.0.smithi181.stdout:399: done (1 left) 2024-09-17T16:02:52.593 INFO:tasks.rados.rados.0.smithi181.stdout: waiting on 1 2024-09-17T16:02:52.603 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:52.601+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:52.606 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing copy_from racing read to smithi18136759-8 2024-09-17T16:02:52.608 INFO:tasks.rados.rados.0.smithi181.stdout:398: finishing copy_from to smithi18136759-8 2024-09-17T16:02:52.608 INFO:tasks.rados.rados.0.smithi181.stdout:update_object_version oid 8 v 53 (ObjNum 117 snap 29 seq_num 117) dirty exists 2024-09-17T16:02:52.608 INFO:tasks.rados.rados.0.smithi181.stdout:398: done (0 left) 2024-09-17T16:02:52.611 INFO:tasks.rados.rados.0.smithi181.stderr:0 errors. 2024-09-17T16:02:52.611 INFO:tasks.rados.rados.0.smithi181.stderr: 2024-09-17T16:02:52.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:02:52.620 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.621 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [], 2024-09-17T16:02:52.621 INFO:teuthology.orchestra.run.smithi169.stdout: "complaint_time": 30, 2024-09-17T16:02:52.621 INFO:teuthology.orchestra.run.smithi169.stdout: "num_blocked_ops": 0 2024-09-17T16:02:52.621 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.629 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:52.704 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:52.702+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:52.732 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2024-09-17T16:02:52.732 INFO:teuthology.orchestra.run.smithi169.stdout: "size": 20, 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 600, 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "ops": [ 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:9 3.0s0 3:03e99693:::smithi18136759-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.877129866000004, 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027474124999999999, 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.733 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 9 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852783+0000", 2024-09-17T16:02:52.734 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852784+0000", 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.9200000000000001e-07 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852788+0000", 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9700000000000001e-06 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.735 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852796+0000", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6879999999999999e-06 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852803+0000", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-06 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855607+0000", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0028037589999999999 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:52.736 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855689+0000", 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.2206999999999994e-05 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.863960+0000", 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0082704949999999992 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.864025+0000", 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5493999999999993e-05 2024-09-17T16:02:52.737 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880257+0000", 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.016232369999999999 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:10 3.0s0 3:046722a1:::smithi18136759-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.877101353, 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.024952149999999999, 2024-09-17T16:02:52.738 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 10 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.739 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852812+0000", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852813+0000", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9699999999999995e-07 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852916+0000", 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000103032 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.740 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852926+0000", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.0428000000000001e-05 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852935+0000", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.855966+0000", 2024-09-17T16:02:52.741 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003030863 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856112+0000", 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000146036 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.856885+0000", 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077235300000000004 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.742 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.862959+0000", 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0060742820000000003 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877690+0000", 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.014731365 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.877764+0000", 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3657000000000003e-05 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.743 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:11 3.0s0 3:046722a1:::smithi18136759-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.876962308000003, 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.027062019999999999, 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.744 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 11 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852951+0000", 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.745 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852952+0000", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.2200000000000003e-07 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852956+0000", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.189e-06 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852964+0000", 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.7889999999999995e-06 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.746 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852969+0000", 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4609999999999999e-06 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858151+0000", 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0051822669999999999 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.747 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858208+0000", 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6665e-05 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.858778+0000", 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00056975000000000005 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879127+0000", 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020348806000000001 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.748 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.879228+0000", 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000101436 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880013+0000", 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078493499999999995 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.749 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:12 3.0s0 3:046722a1:::smithi18136759-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.876934633999994, 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030600270999999998, 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 12 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.750 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852979+0000", 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.6000000000000003e-07 2024-09-17T16:02:52.751 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852983+0000", 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4290000000000001e-06 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852990+0000", 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0999999999999998e-06 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.752 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.852995+0000", 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.4700000000000001e-06 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859153+0000", 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0061573130000000002 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.859236+0000", 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.753 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880364+0000", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.021128088 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.880437+0000", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.3435000000000004e-05 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.883579+0000", 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003141624 2024-09-17T16:02:52.754 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:95 3.9s0 3:9b2471fe:::smithi18136759-32:head [setxattr _header (56) in=7b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.812498055999995, 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033612626999999999, 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.755 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 95 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917415+0000", 2024-09-17T16:02:52.756 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917416+0000", 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.9599999999999999e-07 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917419+0000", 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6100000000000002e-06 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.757 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917426+0000", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.849e-06 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917432+0000", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.3449999999999996e-06 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917560+0000", 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00012873299999999999 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.758 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917657+0000", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.6817000000000005e-05 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918094+0000", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00043698100000000002 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918585+0000", 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00049115599999999997 2024-09-17T16:02:52.759 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950966+0000", 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032380817999999999 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951028+0000", 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.1622000000000001e-05 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.760 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:96 3.9s0 3:9b2471fe:::smithi18136759-32:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.812472851999999, 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.03449402, 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 96 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.761 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917440+0000", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917441+0000", 2024-09-17T16:02:52.762 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.7299999999999996e-07 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917444+0000", 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.762e-06 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917450+0000", 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.618e-06 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.763 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.917455+0000", 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7890000000000002e-06 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918324+0000", 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00086925200000000005 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.918414+0000", 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.9677000000000002e-05 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.764 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951064+0000", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.032649869999999998 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951149+0000", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.5298999999999998e-05 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951934+0000", 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00078518000000000001 2024-09-17T16:02:52.765 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=16, last_complete=35'4, committed=1, applied=1))", 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.809106858000007, 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030231454000000001, 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.766 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.767 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:52.767 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.767 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920806+0000", 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.768 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920807+0000", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.5499999999999997e-07 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.920811+0000", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.2259999999999999e-06 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950888+0000", 2024-09-17T16:02:52.769 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030076418000000001 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950891+0000", 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.5540000000000002e-06 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950921+0000", 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.0091999999999999e-05 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.770 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951038+0000", 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000116409 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:125 3.9s0 3:99c5a4ac:::smithi18136759-42:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.782132208999997, 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.033976998000000001, 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.771 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 125 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947781+0000", 2024-09-17T16:02:52.772 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947782+0000", 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0500000000000003e-07 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947785+0000", 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.9270000000000001e-06 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.773 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947791+0000", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2190000000000001e-06 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947794+0000", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2399999999999999e-06 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949472+0000", 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.001677726 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.774 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949551+0000", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.8998000000000005e-05 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.949873+0000", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000321892 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950604+0000", 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00073159699999999995 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.775 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981659+0000", 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031054702 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981758+0000", 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.9092000000000006e-05 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.776 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:126 3.9s0 3:99c5a4ac:::smithi18136759-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.782114338, 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.036296615999999997, 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 126 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.777 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947799+0000", 2024-09-17T16:02:52.778 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.6399999999999998e-07 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947800+0000", 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.221e-06 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947804+0000", 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.6679999999999999e-06 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.779 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.947807+0000", 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.881e-06 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950057+0000", 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0022504909999999999 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.950105+0000", 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7682999999999997e-05 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.780 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981882+0000", 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.031776999 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981975+0000", 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.3306000000000004e-05 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.781 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.984095+0000", 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0021201029999999999 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 35/32 ECSubWriteReply(tid=18, last_complete=35'6, committed=1, applied=1))", 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 78.778820570999997, 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.030678721999999999, 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.782 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951093+0000", 2024-09-17T16:02:52.783 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.4900000000000001e-07 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.951095+0000", 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.3e-06 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980893+0000", 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02979805 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.784 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.980898+0000", 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4560000000000002e-06 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981591+0000", 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00069284000000000001 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:33.981771+0000", 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00018072699999999999 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.785 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 44/32 ECSubWriteReply(tid=43, last_complete=44'15, committed=1, applied=1))", 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 69.778704816000001, 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72901534199999996, 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.786 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951208+0000", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951209+0000", 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.7000000000000004e-07 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.787 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:42.951213+0000", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9419999999999997e-06 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679938+0000", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.72872537299999995 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.679954+0000", 2024-09-17T16:02:52.788 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5150000000000001e-05 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680104+0000", 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.000150307 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:43.680224+0000", 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0001199 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.789 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 46/32 ECSubWriteReply(tid=50, last_complete=46'21, committed=1, applied=1))", 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 67.798851436000007, 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.148135453, 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.790 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931062+0000", 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.3399999999999999e-07 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.791 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:44.931066+0000", 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.3469999999999999e-06 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078748+0000", 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.147682181 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.792 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078758+0000", 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5780000000000005e-06 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.078813+0000", 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5223000000000003e-05 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:45.079197+0000", 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00038449000000000003 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.793 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.10s0 55/53 ECSubWriteReply(tid=63, last_complete=55'29, committed=1, applied=1))", 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 59.352936196999998, 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.850069713, 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.794 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376977+0000", 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376978+0000", 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.6199999999999998e-07 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.795 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.376981+0000", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.1540000000000002e-06 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226831+0000", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.84985070900000004 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226846+0000", 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.4192999999999999e-05 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.796 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.226976+0000", 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013074699999999999 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:54.227047+0000", 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0347999999999998e-05 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.797 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWrite(3.ds2 55/32 ECSubWrite(tid=102, reqid=client.4560.0:358, at_version=55'50, trim_to=0'0, roll_forward_to=55'47))", 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 59.336736701, 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.023393622999999999, 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.798 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393177+0000", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.82e-07 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.393970+0000", 2024-09-17T16:02:52.799 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00079326400000000001 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414706+0000", 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.020736065000000001 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414712+0000", 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.1390000000000003e-06 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.800 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414788+0000", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.6206999999999997e-05 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.414798+0000", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.8810000000000008e-06 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416538+0000", 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00174007 2024-09-17T16:02:52.801 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:01:53.416570+0000", 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.2415000000000001e-05 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpReadReply(3.0s0 68/53 ECSubReadReply(tid=92, attrs_read=0))", 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 45.871589426, 2024-09-17T16:02:52.802 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.495310574, 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.803 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.858324+0000", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:06.859357+0000", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0010325270000000001 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352866+0000", 2024-09-17T16:02:52.804 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.49350909799999998 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352872+0000", 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.5819999999999998e-06 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.352920+0000", 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.7234999999999998e-05 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.805 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.353634+0000", 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00071472599999999995 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:497 3.9s0 3:905994b3:::smithi18136759-13:head [append 0~0] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 45.356393019999999, 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.035534871000000003, 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.806 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 497 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.807 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373520+0000", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373521+0000", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.8200000000000001e-07 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373523+0000", 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.272e-06 2024-09-17T16:02:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373529+0000", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.2750000000000001e-06 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.373534+0000", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.0359999999999997e-06 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375754+0000", 2024-09-17T16:02:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.002219834 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.375866+0000", 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011218900000000001 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_started", 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.376638+0000", 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00077170400000000001 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "sub_op_committed", 2024-09-17T16:02:52.810 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380515+0000", 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.0038770889999999998 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "commit_sent", 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408971+0000", 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.02845599 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409055+0000", 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 8.4099999999999998e-05 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.811 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 69/53 ECSubWriteReply(tid=97, last_complete=69'23, committed=1, applied=1))", 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 45.349649079999999, 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.029135165000000001, 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.812 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380264+0000", 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.3300000000000001e-07 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.813 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.380265+0000", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.156e-06 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408819+0000", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.028553388999999998 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408826+0000", 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 7.0140000000000001e-06 2024-09-17T16:02:52.814 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.408882+0000", 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 5.5704000000000001e-05 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.409399+0000", 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00051776899999999995 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.815 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 69/53 ECSubWriteReply(tid=104, last_complete=69'50, committed=1, applied=1))", 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 45.310178614000002, 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35452754399999997, 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.816 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419735+0000", 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.6199999999999998e-07 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.817 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.419738+0000", 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.65e-06 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774110+0000", 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.35437182699999997 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.818 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774122+0000", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.2642e-05 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774218+0000", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 9.5758e-05 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:07.774262+0000", 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.4205000000000001e-05 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.819 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 74/53 ECSubWriteReply(tid=138, last_complete=74'39, committed=1, applied=1))", 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 40.325316483000002, 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36342225099999997, 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "reached pg", 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.820 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404597+0000", 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.8000000000000002e-07 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.821 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.404599+0000", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.7549999999999999e-06 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767461+0000", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.36286232000000002 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767477+0000", 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 1.5934e-05 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.822 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.767592+0000", 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00011500199999999999 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:12.768019+0000", 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00042695999999999999 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "description": "osd_op(client.4560.0:763 3.19s0 3:9b2471fe:::smithi18136759-32:10 [read 471893~52395,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-17T16:02:52.823 INFO:teuthology.orchestra.run.smithi169.stdout: "initiated_at": "2024-09-17T16:02:24.459000+0000", 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "age": 28.270913405999998, 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43519143399999999, 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "continuous": false, 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "type_data": { 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "flag_point": "started", 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "client_info": { 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "client": "client.4560", 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "tid": 763 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "events": [ 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "initiated", 2024-09-17T16:02:52.824 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459000+0000", 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "header_read", 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459000+0000", 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "throttled", 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459000+0000", 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 3.9799999999999999e-07 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.825 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "all_read", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459003+0000", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 2.514e-06 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "dispatched", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459008+0000", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.8180000000000003e-06 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "queued_for_pg", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459012+0000", 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 4.532e-06 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.826 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459149+0000", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00013716800000000001 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "waiting_for_map not empty", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.459155+0000", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 6.0090000000000004e-06 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "reached_pg", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.462601+0000", 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.003446232 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.827 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "started", 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.462702+0000", 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.00010046499999999999 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: }, 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: { 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "event": "done", 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "time": "2024-09-17T16:02:24.894191+0000", 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: "duration": 0.43148929800000002 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: } 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2024-09-17T16:02:52.828 INFO:teuthology.orchestra.run.smithi169.stdout:} 2024-09-17T16:02:52.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:52.831 INFO:tasks.ceph.osd.11.smithi181.stderr:2024-09-17T16:02:52.828+0000 7f76678bf640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 33267) UID: 0 2024-09-17T16:02:52.931 INFO:tasks.ceph.osd.5.smithi050.stderr:2024-09-17T16:02:52.929+0000 7f4407d26640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33108) UID: 0 2024-09-17T16:02:52.945 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:52.945 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:52.945 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:52.945 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:52.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:52.959 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:52.959 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":114,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:02:52.576895+0000","last_up_change":"2024-09-17T16:02:03.333624+0000","last_in_change":"2024-09-17T16:01:06.794253+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-17T16:01:29.829066+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":19,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"53","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":114,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":57,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6819","nonce":3415295348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6821","nonce":3415295348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6825","nonce":3415295348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3415295348},{"type":"v1","addr":"172.21.15.181:6823","nonce":3415295348}]},"public_addr":"172.21.15.181:6819/3415295348","cluster_addr":"172.21.15.181:6821/3415295348","heartbeat_back_addr":"172.21.15.181:6825/3415295348","heartbeat_front_addr":"172.21.15.181:6823/3415295348","state":["exists"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":0.785186767578125,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:32.312761+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:31.399250+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:30.409412+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:41.432577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:25.617972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:25.371442+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:32.431042+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:01:13.664321+0000","dead_epoch":65},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:26.141597+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:25.015167+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:38.091678+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:35.472667+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:41.182127+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:39.544307+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:31.549706+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:26.664633+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[6,11]},{"pgid":"2.7","osds":[13,15]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":8,"length":3},{"begin":13,"length":4},{"begin":20,"length":2},{"begin":23,"length":2},{"begin":26,"length":8},{"begin":35,"length":2},{"begin":39,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:02:52.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:02:53.032 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:53.030+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:53.056 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.056 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:53.056 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:53.056 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:53.056 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.065 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:53.133 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:53.131+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.168 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 44/32 ECSubWrite(tid=167, reqid=client.4560.0:228, at_version=44'14, trim_to=0'0, roll_forward_to=43'10))", 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 70.257152894000001, 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045964070000000003, 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.169 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908481+0000", 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908482+0000", 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.017e-06 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.170 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908499+0000", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6748999999999998e-05 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908533+0000", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3831999999999997e-05 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908539+0000", 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5980000000000002e-06 2024-09-17T16:02:53.171 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908619+0000", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9740999999999994e-05 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.908629+0000", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4369999999999996e-06 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913043+0000", 2024-09-17T16:02:53.172 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044149380000000002 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:42.913078+0000", 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4094999999999998e-05 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=177, reqid=client.4560.0:268, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:53.173 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.237889713000001, 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046997530000000001, 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927744+0000", 2024-09-17T16:02:53.174 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927762+0000", 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7229000000000001e-05 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927945+0000", 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00018362200000000001 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.175 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928161+0000", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00021584100000000001 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928170+0000", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6249999999999996e-06 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928242+0000", 2024-09-17T16:02:53.176 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2434999999999993e-05 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928253+0000", 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0890000000000001e-05 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932409+0000", 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041564169999999999 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.177 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932444+0000", 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4693999999999997e-05 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=178, reqid=client.4560.0:269, at_version=45'17, trim_to=0'0, roll_forward_to=44'14))", 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.237684686999998, 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048629329999999998, 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.178 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927949+0000", 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.179 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.927950+0000", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.3399999999999999e-07 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928041+0000", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0953999999999997e-05 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928205+0000", 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000164537 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.180 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928213+0000", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.272e-06 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929483+0000", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001270106 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929493+0000", 2024-09-17T16:02:53.181 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0207000000000001e-05 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932778+0000", 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032847419999999998 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932812+0000", 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.4581e-05 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.182 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=12, reqid=client.4560.0:273, at_version=45'11, trim_to=0'0, roll_forward_to=44'8))", 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.237553168000005, 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043524990000000001, 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.183 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928081+0000", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928082+0000", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.2799999999999996e-07 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928098+0000", 2024-09-17T16:02:53.184 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6070999999999999e-05 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928229+0000", 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013130999999999999 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928234+0000", 2024-09-17T16:02:53.185 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4879999999999998e-06 2024-09-17T16:02:53.186 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.186 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.186 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.186 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928583+0000", 2024-09-17T16:02:53.186 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034869300000000002 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.928595+0000", 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1433e-05 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.189 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932398+0000", 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038034890000000002 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.932433+0000", 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5386999999999998e-05 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 46/32 ECSubWrite(tid=47, reqid=osd.14.0:47, at_version=46'19, trim_to=0'0, roll_forward_to=44'15))", 2024-09-17T16:02:53.190 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 68.246036484000001, 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043836329999999996, 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.191 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919598+0000", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6899999999999996e-07 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919621+0000", 2024-09-17T16:02:53.192 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2204e-05 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919652+0000", 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1846e-05 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919661+0000", 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4020000000000002e-06 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.193 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919724+0000", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3075000000000002e-05 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919733+0000", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.3540000000000006e-06 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923937+0000", 2024-09-17T16:02:53.194 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042032980000000003 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923981+0000", 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4685000000000003e-05 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 52/32 ECSubWrite(tid=79, reqid=client.4560.0:332, at_version=52'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 62.808019655999999, 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051054819999999997, 2024-09-17T16:02:53.195 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357614+0000", 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.196 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.357615+0000", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.37e-07 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358043+0000", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042830599999999999 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358075+0000", 2024-09-17T16:02:53.197 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2512999999999999e-05 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358085+0000", 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0081000000000001e-05 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358170+0000", 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4362000000000003e-05 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.198 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.358181+0000", 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1535000000000001e-05 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362680+0000", 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989069999999999 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:50.362720+0000", 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9641000000000002e-05 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.199 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=197, reqid=client.4560.0:392, at_version=60'28, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.815262148000002, 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0099856809999999997, 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.200 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350372+0000", 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2599999999999999e-07 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.201 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350537+0000", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016478400000000001 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350604+0000", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7305999999999996e-05 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350615+0000", 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1154e-05 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.202 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350701+0000", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6136000000000003e-05 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350721+0000", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9967e-05 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360314+0000", 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0095931230000000003 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.203 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360358+0000", 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3084999999999998e-05 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=198, reqid=client.4560.0:393, at_version=60'29, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.812736432000001, 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075206600000000002, 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.204 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352898+0000", 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.205 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.352899+0000", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1230000000000001e-06 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353041+0000", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014228299999999999 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353091+0000", 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9942999999999999e-05 2024-09-17T16:02:53.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353101+0000", 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0448999999999999e-05 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353167+0000", 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5365999999999997e-05 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.207 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.353178+0000", 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1429e-05 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360364+0000", 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071857500000000003 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360418+0000", 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4317000000000002e-05 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.208 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=199, reqid=client.4560.0:394, at_version=60'30, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.811047881, 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058628329999999996, 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:53.209 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354586+0000", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354587+0000", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4099999999999997e-07 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354652+0000", 2024-09-17T16:02:53.210 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5085000000000005e-05 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354702+0000", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9641999999999997e-05 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354711+0000", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0869999999999996e-06 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354774+0000", 2024-09-17T16:02:53.211 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3048999999999999e-05 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.354784+0000", 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0163000000000001e-05 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360427+0000", 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056429729999999999 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.212 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360449+0000", 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1993000000000001e-05 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 60/53 ECSubWrite(tid=200, reqid=client.4560.0:395, at_version=60'31, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 54.809621741999997, 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044989890000000001, 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.213 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356012+0000", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356013+0000", 2024-09-17T16:02:53.214 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9100000000000003e-07 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356031+0000", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7592999999999999e-05 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356068+0000", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7345999999999998e-05 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356079+0000", 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0730999999999999e-05 2024-09-17T16:02:53.215 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356150+0000", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1668999999999994e-05 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356161+0000", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1027e-05 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360478+0000", 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043165800000000004 2024-09-17T16:02:53.216 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360511+0000", 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3251999999999999e-05 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=131, reqid=osd.13.0:131, at_version=64'15, trim_to=0'0, roll_forward_to=62'13))", 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 50.827378029000002, 2024-09-17T16:02:53.217 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046971620000000004, 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.218 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338256+0000", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0600000000000001e-07 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338276+0000", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.9932000000000002e-05 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338298+0000", 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1336e-05 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.219 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338306+0000", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6810000000000004e-06 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338467+0000", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000160538 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.338477+0000", 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5929999999999995e-06 2024-09-17T16:02:53.220 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342913+0000", 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044366149999999997 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.342953+0000", 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0061000000000002e-05 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.221 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s2 64/53 ECSubWrite(tid=134, reqid=osd.13.0:131, at_version=0'0, trim_to=0'0, roll_forward_to=64'15))", 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 50.815587268999998, 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042510439999999998, 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.222 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350047+0000", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0900000000000001e-07 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350056+0000", 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4880000000000007e-06 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.223 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350080+0000", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4360999999999999e-05 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350089+0000", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2609999999999993e-06 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350189+0000", 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000100868 2024-09-17T16:02:53.224 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.350199+0000", 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1679999999999993e-06 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354267+0000", 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040685629999999999 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.225 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354298+0000", 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0626000000000001e-05 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.2s5 64/53 ECSubWrite(tid=211, reqid=osd.11.0:211, at_version=64'37, trim_to=0'0, roll_forward_to=62'35))", 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 50.806567897000001, 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048593489999999998, 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.226 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359066+0000", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.227 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359067+0000", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6699999999999999e-07 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359087+0000", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0594999999999998e-05 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359129+0000", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1458999999999997e-05 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.228 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359138+0000", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6050000000000004e-06 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359207+0000", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.8885000000000002e-05 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.359218+0000", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0524e-05 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.229 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363901+0000", 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046832100000000002 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.363925+0000", 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4703999999999999e-05 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=93, reqid=osd.14.0:93, at_version=69'46, trim_to=0'0, roll_forward_to=68'44))", 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:53.230 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.814609249999997, 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059954810000000004, 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:53.231 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351025+0000", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6299999999999999e-07 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351050+0000", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4579e-05 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351200+0000", 2024-09-17T16:02:53.232 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015020800000000001 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.351208+0000", 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6799999999999999e-06 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352434+0000", 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012252350000000001 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352445+0000", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1440000000000001e-05 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.356973+0000", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0045277900000000003 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.357020+0000", 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.234 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 69/53 ECSubWrite(tid=102, reqid=client.4560.0:502, at_version=69'50, trim_to=0'0, roll_forward_to=69'46))", 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 45.755628262000002, 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042988599999999998, 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.235 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410006+0000", 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.37e-07 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.236 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410021+0000", 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4741e-05 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410042+0000", 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0798e-05 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410047+0000", 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5459999999999999e-06 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.237 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410084+0000", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6556e-05 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.410089+0000", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4600000000000002e-06 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414267+0000", 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0041775720000000001 2024-09-17T16:02:53.238 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.414305+0000", 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.765e-05 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 81/53 ECSubWrite(tid=151, reqid=osd.14.0:151, at_version=81'76, trim_to=0'0, roll_forward_to=77'72))", 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 33.726110560000002, 2024-09-17T16:02:53.239 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048353479999999997, 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.240 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439523+0000", 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439524+0000", 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7000000000000004e-07 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439546+0000", 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1430999999999999e-05 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.241 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439578+0000", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1962000000000003e-05 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439590+0000", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2201999999999999e-05 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439662+0000", 2024-09-17T16:02:53.242 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2361999999999996e-05 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439675+0000", 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2848e-05 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444304+0000", 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046291250000000004 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.243 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444359+0000", 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4648000000000003e-05 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.14s3 81/53 ECSubWrite(tid=39, reqid=osd.1.0:38, at_version=0'0, trim_to=0'0, roll_forward_to=81'30))", 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 33.725870106000002, 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0046130629999999997, 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.244 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439764+0000", 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.245 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439765+0000", 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.001e-06 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439778+0000", 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3295999999999999e-05 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.246 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439815+0000", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6511000000000003e-05 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439828+0000", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.329e-05 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439899+0000", 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1125999999999996e-05 2024-09-17T16:02:53.247 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.439911+0000", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1792e-05 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444330+0000", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044186609999999999 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.444377+0000", 2024-09-17T16:02:53.248 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7386000000000003e-05 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=241, reqid=client.4560.0:860, at_version=104'100, trim_to=0'0, roll_forward_to=102'98))", 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.624842706000001, 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0051699650000000003, 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.249 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540791+0000", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540792+0000", 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-17T16:02:53.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540823+0000", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1467999999999998e-05 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540874+0000", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0460999999999997e-05 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540884+0000", 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0832999999999999e-05 2024-09-17T16:02:53.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540957+0000", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3083000000000005e-05 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.540969+0000", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1712000000000001e-05 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545929+0000", 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049599020000000004 2024-09-17T16:02:53.252 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545961+0000", 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2221999999999997e-05 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=242, reqid=client.4560.0:861, at_version=104'101, trim_to=0'0, roll_forward_to=102'98))", 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.624478972, 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0048239809999999998, 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.253 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541155+0000", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.254 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541156+0000", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7400000000000003e-07 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541183+0000", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7566999999999999e-05 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541215+0000", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2101000000000003e-05 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541223+0000", 2024-09-17T16:02:53.255 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1000000000000004e-06 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541924+0000", 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00070091800000000003 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.541935+0000", 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0835999999999999e-05 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.256 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545966+0000", 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040307620000000002 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.545979+0000", 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3123e-05 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.0s3 104/53 ECSubWrite(tid=245, reqid=client.4560.0:862, at_version=0'0, trim_to=0'0, roll_forward_to=104'102))", 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 10.618168636, 2024-09-17T16:02:53.257 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.63482562300000001, 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547465+0000", 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.258 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547466+0000", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0759999999999999e-06 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:42.547476+0000", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.6709999999999994e-06 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181369+0000", 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.63389281099999994 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.259 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181396+0000", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.741e-05 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181516+0000", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000120091 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.181539+0000", 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2106000000000001e-05 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.260 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.182254+0000", 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071494500000000003 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:43.182291+0000", 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7512999999999998e-05 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.261 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:53.263 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:53.261+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:53.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:02:53.318 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:02:53.329 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-17T16:02:52.958881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131908147,"num_objects":169,"num_object_clones":115,"num_object_copies":1163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":9,"num_read":393,"num_read_kb":124583,"num_write":725,"num_write_kb":116690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1685,"ondisk_log_size":1685,"up":200,"acting":200,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":201,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":718088,"kb_used_data":227336,"kb_used_omap":24,"kb_used_meta":490343,"kb_avail":1509231352,"statfs":{"total":1546188226560,"available":1545452904448,"internally_reserved":0,"allocated":232792064,"data_stored":338024514,"data_compressed":4213198,"data_compressed_allocated":5656576,"data_compressed_original":12189696,"omap_allocated":25519,"internal_metadata":502111313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9076736,"num_objects":4,"num_object_clones":4,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-1,"num_read":13,"num_read_kb":7176,"num_write":37,"num_write_kb":8864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001156"},"pg_stats":[{"pgid":"3.19","version":"107'61","reported_seq":500,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586818+0000","last_change":"2024-09-17T16:02:45.548982+0000","last_active":"2024-09-17T16:02:49.586818+0000","last_peered":"2024-09-17T16:02:49.586818+0000","last_clean":"2024-09-17T16:02:49.586818+0000","last_became_active":"2024-09-17T16:02:36.523148+0000","last_became_peered":"2024-09-17T16:02:36.523148+0000","last_unstale":"2024-09-17T16:02:49.586818+0000","last_undegraded":"2024-09-17T16:02:49.586818+0000","last_fullsized":"2024-09-17T16:02:49.586818+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:01:59.252035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.252035+0000","objects_scrubbed":4,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:19.792768+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0078799190000000009,"stat_sum":{"num_bytes":6275072,"num_objects":5,"num_object_clones":3,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":4719,"num_write":10,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8945664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,1],"acting":[14,11,13,15,4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.18","version":"107'68","reported_seq":480,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686756+0000","last_change":"2024-09-17T16:02:45.553659+0000","last_active":"2024-09-17T16:02:49.686756+0000","last_peered":"2024-09-17T16:02:49.686756+0000","last_clean":"2024-09-17T16:02:49.686756+0000","last_became_active":"2024-09-17T16:01:53.177881+0000","last_became_peered":"2024-09-17T16:01:53.177881+0000","last_unstale":"2024-09-17T16:02:49.686756+0000","last_undegraded":"2024-09-17T16:02:49.686756+0000","last_fullsized":"2024-09-17T16:02:49.686756+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'31","last_scrub_stamp":"2024-09-17T16:02:00.137409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.137409+0000","objects_scrubbed":2,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:17.850651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0088981600000000004,"stat_sum":{"num_bytes":5603328,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":19990,"num_write":13,"num_write_kb":4736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.17","version":"84'44","reported_seq":365,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687028+0000","last_change":"2024-09-17T16:02:04.352378+0000","last_active":"2024-09-17T16:02:49.687028+0000","last_peered":"2024-09-17T16:02:49.687028+0000","last_clean":"2024-09-17T16:02:02.351935+0000","last_became_active":"2024-09-17T16:02:04.352378+0000","last_became_peered":"2024-09-17T16:02:04.352378+0000","last_unstale":"2024-09-17T16:02:49.687028+0000","last_undegraded":"2024-09-17T16:02:04.343159+0000","last_fullsized":"2024-09-17T16:02:04.342807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'27","last_scrub_stamp":"2024-09-17T16:02:02.077678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.077678+0000","objects_scrubbed":3,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:20.268681+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.027309317,"stat_sum":{"num_bytes":5914624,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":6628,"num_write":11,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.16","version":"88'38","reported_seq":366,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687939+0000","last_change":"2024-09-17T16:02:04.351696+0000","last_active":"2024-09-17T16:02:49.687939+0000","last_peered":"2024-09-17T16:02:49.687939+0000","last_clean":"2024-09-17T16:02:02.354687+0000","last_became_active":"2024-09-17T16:02:04.351696+0000","last_became_peered":"2024-09-17T16:02:04.351696+0000","last_unstale":"2024-09-17T16:02:49.687939+0000","last_undegraded":"2024-09-17T16:02:04.341016+0000","last_fullsized":"2024-09-17T16:02:04.340608+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'16","last_scrub_stamp":"2024-09-17T16:02:00.050173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.050173+0000","objects_scrubbed":4,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:04.066075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.01423026,"stat_sum":{"num_bytes":6881280,"num_objects":9,"num_object_clones":7,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":8,"num_read_kb":7052,"num_write":12,"num_write_kb":3232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":9}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.15","version":"107'40","reported_seq":452,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586193+0000","last_change":"2024-09-17T16:02:45.562350+0000","last_active":"2024-09-17T16:02:49.586193+0000","last_peered":"2024-09-17T16:02:49.586193+0000","last_clean":"2024-09-17T16:02:49.586193+0000","last_became_active":"2024-09-17T16:01:53.142297+0000","last_became_peered":"2024-09-17T16:01:53.142297+0000","last_unstale":"2024-09-17T16:02:49.586193+0000","last_undegraded":"2024-09-17T16:02:49.586193+0000","last_fullsized":"2024-09-17T16:02:49.586193+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'8","last_scrub_stamp":"2024-09-17T16:01:58.940426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.940426+0000","objects_scrubbed":2,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:02.151901+0000","scrub_duration":453,"objects_trimmed":0,"snaptrim_duration":0.012200225,"stat_sum":{"num_bytes":2834432,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":2000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.14","version":"107'36","reported_seq":450,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-17T16:02:48.572679+0000","last_change":"2024-09-17T16:02:45.550997+0000","last_active":"2024-09-17T16:02:48.572679+0000","last_peered":"2024-09-17T16:02:48.572679+0000","last_clean":"2024-09-17T16:02:48.572679+0000","last_became_active":"2024-09-17T16:01:52.360122+0000","last_became_peered":"2024-09-17T16:01:52.360122+0000","last_unstale":"2024-09-17T16:02:48.572679+0000","last_undegraded":"2024-09-17T16:02:48.572679+0000","last_fullsized":"2024-09-17T16:02:48.572679+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:58.496684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.496684+0000","objects_scrubbed":4,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:03.622659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0092466449999999995,"stat_sum":{"num_bytes":1064960,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":1185,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686418+0000","last_change":"2024-09-17T16:01:58.347762+0000","last_active":"2024-09-17T16:02:49.686418+0000","last_peered":"2024-09-17T16:02:49.686418+0000","last_clean":"2024-09-17T16:02:49.686418+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:02:49.686418+0000","last_undegraded":"2024-09-17T16:02:49.686418+0000","last_fullsized":"2024-09-17T16:02:49.686418+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:34.031282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.6","version":"85'24","reported_seq":345,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.688197+0000","last_change":"2024-09-17T16:02:04.349120+0000","last_active":"2024-09-17T16:02:49.688197+0000","last_peered":"2024-09-17T16:02:49.688197+0000","last_clean":"2024-09-17T16:02:02.355614+0000","last_became_active":"2024-09-17T16:02:04.349120+0000","last_became_peered":"2024-09-17T16:02:04.349120+0000","last_unstale":"2024-09-17T16:02:49.688197+0000","last_undegraded":"2024-09-17T16:02:04.340835+0000","last_fullsized":"2024-09-17T16:02:04.340587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:02:01.398192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.398192+0000","objects_scrubbed":1,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:24.186661+0000","scrub_duration":371,"objects_trimmed":0,"snaptrim_duration":0.018760373,"stat_sum":{"num_bytes":3096576,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":964,"num_write":9,"num_write_kb":2320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2147483647,15,8,11,0,10],"acting":[13,2147483647,15,8,11,0,10],"avail_no_missing":["13(0)","0(5)","8(3)","10(6)","11(4)","15(2)"],"object_location_counts":[{"shards":"0(5),8(3),10(6),11(4),13(0),15(2)","objects":4}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":193,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.566651+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:02:49.566651+0000","last_peered":"2024-09-17T16:02:49.566651+0000","last_clean":"2024-09-17T16:02:49.566651+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:02:49.566651+0000","last_undegraded":"2024-09-17T16:02:49.566651+0000","last_fullsized":"2024-09-17T16:02:49.566651+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"72'22","reported_seq":329,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687742+0000","last_change":"2024-09-17T16:02:04.349605+0000","last_active":"2024-09-17T16:02:49.687742+0000","last_peered":"2024-09-17T16:02:49.687742+0000","last_clean":"2024-09-17T16:02:03.036314+0000","last_became_active":"2024-09-17T16:02:04.349605+0000","last_became_peered":"2024-09-17T16:02:04.349605+0000","last_unstale":"2024-09-17T16:02:49.687742+0000","last_undegraded":"2024-09-17T16:02:04.340916+0000","last_fullsized":"2024-09-17T16:02:04.340610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01416717,"stat_sum":{"num_bytes":1261568,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":3,"num_read_kb":2,"num_write":15,"num_write_kb":1232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,6,2147483647,1,11,5,0],"acting":[13,6,2147483647,1,11,5,0],"avail_no_missing":["13(0)","0(6)","1(3)","5(5)","6(1)","11(4)"],"object_location_counts":[{"shards":"0(6),1(3),5(5),6(1),11(4),13(0)","objects":3}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":185,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586733+0000","last_change":"2024-09-17T16:01:20.762413+0000","last_active":"2024-09-17T16:02:49.586733+0000","last_peered":"2024-09-17T16:02:49.586733+0000","last_clean":"2024-09-17T16:02:49.586733+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:02:49.586733+0000","last_undegraded":"2024-09-17T16:02:49.586733+0000","last_fullsized":"2024-09-17T16:02:49.586733+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:19.517195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"48'16","reported_seq":414,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-17T16:02:48.572958+0000","last_change":"2024-09-17T16:02:45.545305+0000","last_active":"2024-09-17T16:02:48.572958+0000","last_peered":"2024-09-17T16:02:48.572958+0000","last_clean":"2024-09-17T16:02:48.572958+0000","last_became_active":"2024-09-17T16:01:52.351079+0000","last_became_peered":"2024-09-17T16:01:52.351079+0000","last_unstale":"2024-09-17T16:02:48.572958+0000","last_undegraded":"2024-09-17T16:02:48.572958+0000","last_fullsized":"2024-09-17T16:02:48.572958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'16","last_scrub_stamp":"2024-09-17T16:01:59.453974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.453974+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:20.827827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033854190000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,11,3,15,9,13],"acting":[1,2,11,3,15,9,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-17T16:02:48.572900+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:02:48.572900+0000","last_peered":"2024-09-17T16:02:48.572900+0000","last_clean":"2024-09-17T16:02:48.572900+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:02:48.572900+0000","last_undegraded":"2024-09-17T16:02:48.572900+0000","last_fullsized":"2024-09-17T16:02:48.572900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:43.317628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"107'45","reported_seq":449,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586472+0000","last_change":"2024-09-17T16:02:45.563876+0000","last_active":"2024-09-17T16:02:49.586472+0000","last_peered":"2024-09-17T16:02:49.586472+0000","last_clean":"2024-09-17T16:02:49.586472+0000","last_became_active":"2024-09-17T16:01:52.345743+0000","last_became_peered":"2024-09-17T16:01:52.345743+0000","last_unstale":"2024-09-17T16:02:49.586472+0000","last_undegraded":"2024-09-17T16:02:49.586472+0000","last_fullsized":"2024-09-17T16:02:49.586472+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"58'8","last_scrub_stamp":"2024-09-17T16:01:59.457065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.457065+0000","objects_scrubbed":2,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.191844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013638666000000001,"stat_sum":{"num_bytes":344064,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13,5,11,9,1,6],"acting":[8,13,5,11,9,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.3","version":"107'49","reported_seq":459,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686691+0000","last_change":"2024-09-17T16:02:45.560669+0000","last_active":"2024-09-17T16:02:49.686691+0000","last_peered":"2024-09-17T16:02:49.686691+0000","last_clean":"2024-09-17T16:02:49.686691+0000","last_became_active":"2024-09-17T16:01:52.349931+0000","last_became_peered":"2024-09-17T16:01:52.349931+0000","last_unstale":"2024-09-17T16:02:49.686691+0000","last_undegraded":"2024-09-17T16:02:49.686691+0000","last_fullsized":"2024-09-17T16:02:49.686691+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"54'18","last_scrub_stamp":"2024-09-17T16:01:59.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.099258+0000","objects_scrubbed":4,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.860523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015160274,"stat_sum":{"num_bytes":1212416,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":1605,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":196,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586285+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:02:49.586285+0000","last_peered":"2024-09-17T16:02:49.586285+0000","last_clean":"2024-09-17T16:02:49.586285+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:02:49.586285+0000","last_undegraded":"2024-09-17T16:02:49.586285+0000","last_fullsized":"2024-09-17T16:02:49.586285+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.0","version":"107'115","reported_seq":512,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586969+0000","last_change":"2024-09-17T16:02:45.559413+0000","last_active":"2024-09-17T16:02:49.586969+0000","last_peered":"2024-09-17T16:02:49.586969+0000","last_clean":"2024-09-17T16:02:49.586969+0000","last_became_active":"2024-09-17T16:01:52.345493+0000","last_became_peered":"2024-09-17T16:01:52.345493+0000","last_unstale":"2024-09-17T16:02:49.586969+0000","last_undegraded":"2024-09-17T16:02:49.586969+0000","last_fullsized":"2024-09-17T16:02:49.586969+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'27","last_scrub_stamp":"2024-09-17T16:02:00.289587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.289587+0000","objects_scrubbed":4,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:05.455042+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.018264829999999999,"stat_sum":{"num_bytes":7077888,"num_objects":10,"num_object_clones":8,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":16,"num_read_kb":9563,"num_write":13,"num_write_kb":5776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,2,4,3,5,11,9],"acting":[14,2,4,3,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586235+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:02:49.586235+0000","last_peered":"2024-09-17T16:02:49.586235+0000","last_clean":"2024-09-17T16:02:49.586235+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:02:49.586235+0000","last_undegraded":"2024-09-17T16:02:49.586235+0000","last_fullsized":"2024-09-17T16:02:49.586235+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:41.693243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"100'23","reported_seq":343,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687592+0000","last_change":"2024-09-17T16:02:04.352734+0000","last_active":"2024-09-17T16:02:49.687592+0000","last_peered":"2024-09-17T16:02:49.687592+0000","last_clean":"2024-09-17T16:02:02.350917+0000","last_became_active":"2024-09-17T16:02:04.352734+0000","last_became_peered":"2024-09-17T16:02:04.352734+0000","last_unstale":"2024-09-17T16:02:49.687592+0000","last_undegraded":"2024-09-17T16:02:04.341137+0000","last_fullsized":"2024-09-17T16:02:04.340823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.086155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.086155+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:03.525392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.014571756,"stat_sum":{"num_bytes":2260992,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":447,"num_write":24,"num_write_kb":976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,4,3,11,10,2,2147483647],"acting":[13,4,3,11,10,2,2147483647],"avail_no_missing":["13(0)","2(5)","3(2)","4(1)","10(4)","11(3)"],"object_location_counts":[{"shards":"2(5),3(2),4(1),10(4),11(3),13(0)","objects":6}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":194,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.585999+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:02:49.585999+0000","last_peered":"2024-09-17T16:02:49.585999+0000","last_clean":"2024-09-17T16:02:49.585999+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:02:49.585999+0000","last_undegraded":"2024-09-17T16:02:49.585999+0000","last_fullsized":"2024-09-17T16:02:49.585999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.2","version":"97'49","reported_seq":367,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687135+0000","last_change":"2024-09-17T16:02:04.349933+0000","last_active":"2024-09-17T16:02:49.687135+0000","last_peered":"2024-09-17T16:02:49.687135+0000","last_clean":"2024-09-17T16:02:02.354956+0000","last_became_active":"2024-09-17T16:02:04.349933+0000","last_became_peered":"2024-09-17T16:02:04.349933+0000","last_unstale":"2024-09-17T16:02:49.687135+0000","last_undegraded":"2024-09-17T16:02:04.341956+0000","last_fullsized":"2024-09-17T16:02:04.341160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'35","last_scrub_stamp":"2024-09-17T16:02:01.159864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.159864+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:03.447148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023769390000000001,"stat_sum":{"num_bytes":4030464,"num_objects":11,"num_object_clones":8,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":4,"num_read_kb":138,"num_write":25,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,0,4,8,3,2147483647],"acting":[11,2,0,4,8,3,2147483647],"avail_no_missing":["11(0)","0(2)","2(1)","3(5)","4(3)","8(4)"],"object_location_counts":[{"shards":"0(2),2(1),3(5),4(3),8(4),11(0)","objects":11}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":186,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.687076+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:02:49.687076+0000","last_peered":"2024-09-17T16:02:49.687076+0000","last_clean":"2024-09-17T16:02:49.687076+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:02:49.687076+0000","last_undegraded":"2024-09-17T16:02:49.687076+0000","last_fullsized":"2024-09-17T16:02:49.687076+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:41.299345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":328,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686948+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:02:49.686948+0000","last_peered":"2024-09-17T16:02:49.686948+0000","last_clean":"2024-09-17T16:02:49.686948+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:02:49.686948+0000","last_undegraded":"2024-09-17T16:02:49.686948+0000","last_fullsized":"2024-09-17T16:02:49.686948+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:42.104346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.8","version":"107'64","reported_seq":465,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686900+0000","last_change":"2024-09-17T16:02:45.561936+0000","last_active":"2024-09-17T16:02:49.686900+0000","last_peered":"2024-09-17T16:02:49.686900+0000","last_clean":"2024-09-17T16:02:49.686900+0000","last_became_active":"2024-09-17T16:01:52.347600+0000","last_became_peered":"2024-09-17T16:01:52.347600+0000","last_unstale":"2024-09-17T16:02:49.686900+0000","last_undegraded":"2024-09-17T16:02:49.686900+0000","last_fullsized":"2024-09-17T16:02:49.686900+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'33","last_scrub_stamp":"2024-09-17T16:02:02.109565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:02.109565+0000","objects_scrubbed":3,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:10.369530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.016053030999999999,"stat_sum":{"num_bytes":2277376,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":10,"num_read_kb":10525,"num_write":7,"num_write_kb":1744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,2,15,14,0,6],"acting":[11,1,2,15,14,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.9","version":"107'77","reported_seq":498,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.586765+0000","last_change":"2024-09-17T16:02:45.549309+0000","last_active":"2024-09-17T16:02:49.586765+0000","last_peered":"2024-09-17T16:02:49.586765+0000","last_clean":"2024-09-17T16:02:49.586765+0000","last_became_active":"2024-09-17T16:02:36.521024+0000","last_became_peered":"2024-09-17T16:02:36.521024+0000","last_unstale":"2024-09-17T16:02:49.586765+0000","last_undegraded":"2024-09-17T16:02:49.586765+0000","last_fullsized":"2024-09-17T16:02:49.586765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'19","last_scrub_stamp":"2024-09-17T16:02:01.251580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.251580+0000","objects_scrubbed":2,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:23.741835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0082101090000000002,"stat_sum":{"num_bytes":5128192,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":452,"num_write":9,"num_write_kb":4000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5373952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,11,13,15,4,9,1],"acting":[14,11,13,15,4,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.a","version":"111'72","reported_seq":368,"reported_epoch":113,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:51.580707+0000","last_change":"2024-09-17T16:02:04.994580+0000","last_active":"2024-09-17T16:02:51.580707+0000","last_peered":"2024-09-17T16:02:51.580707+0000","last_clean":"2024-09-17T16:02:02.349037+0000","last_became_active":"2024-09-17T16:02:04.994580+0000","last_became_peered":"2024-09-17T16:02:04.994580+0000","last_unstale":"2024-09-17T16:02:51.580707+0000","last_undegraded":"2024-09-17T16:02:04.341211+0000","last_fullsized":"2024-09-17T16:02:04.340891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013081311999999999,"stat_sum":{"num_bytes":15171584,"num_objects":12,"num_object_clones":10,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":20,"num_read_kb":20813,"num_write":41,"num_write_kb":17024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,2147483647,3,14,6,8],"acting":[12,4,2147483647,3,14,6,8],"avail_no_missing":["12(0)","3(3)","4(1)","6(5)","8(6)","14(4)"],"object_location_counts":[{"shards":"3(3),4(1),6(5),8(6),12(0),14(4)","objects":12}],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.b","version":"109'30","reported_seq":333,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.687123+0000","last_change":"2024-09-17T16:02:04.347340+0000","last_active":"2024-09-17T16:02:49.687123+0000","last_peered":"2024-09-17T16:02:49.687123+0000","last_clean":"2024-09-17T16:02:02.337229+0000","last_became_active":"2024-09-17T16:02:04.347340+0000","last_became_peered":"2024-09-17T16:02:04.347340+0000","last_unstale":"2024-09-17T16:02:49.687123+0000","last_undegraded":"2024-09-17T16:02:04.340875+0000","last_fullsized":"2024-09-17T16:02:04.340555+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010428939999999999,"stat_sum":{"num_bytes":5406720,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":22,"num_read_kb":1684,"num_write":18,"num_write_kb":5280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,1,11,15,6,4,2147483647],"acting":[13,1,11,15,6,4,2147483647],"avail_no_missing":["13(0)","1(1)","4(5)","6(4)","11(2)","15(3)"],"object_location_counts":[{"shards":"1(1),4(5),6(4),11(2),13(0),15(3)","objects":7}],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.c","version":"110'60","reported_seq":436,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-17T16:02:48.561529+0000","last_change":"2024-09-17T16:02:45.554035+0000","last_active":"2024-09-17T16:02:48.561529+0000","last_peered":"2024-09-17T16:02:48.561529+0000","last_clean":"2024-09-17T16:02:48.561529+0000","last_became_active":"2024-09-17T16:01:31.979031+0000","last_became_peered":"2024-09-17T16:01:31.979031+0000","last_unstale":"2024-09-17T16:02:48.561529+0000","last_undegraded":"2024-09-17T16:02:48.561529+0000","last_fullsized":"2024-09-17T16:02:48.561529+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'56","last_scrub_stamp":"2024-09-17T16:02:35.300544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.300544+0000","objects_scrubbed":5,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:44.470875+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0013897759999999999,"stat_sum":{"num_bytes":3473408,"num_objects":6,"num_object_clones":4,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":6104,"num_write":57,"num_write_kb":3840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,13,12,10,6,1,9],"acting":[0,13,12,10,6,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.d","version":"112'265","reported_seq":637,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-17T16:02:51.580514+0000","last_change":"2024-09-17T16:02:45.577012+0000","last_active":"2024-09-17T16:02:51.580514+0000","last_peered":"2024-09-17T16:02:51.580514+0000","last_clean":"2024-09-17T16:02:51.580514+0000","last_became_active":"2024-09-17T16:01:32.022196+0000","last_became_peered":"2024-09-17T16:01:32.022196+0000","last_unstale":"2024-09-17T16:02:51.580514+0000","last_undegraded":"2024-09-17T16:02:51.580514+0000","last_fullsized":"2024-09-17T16:02:51.580514+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.025281254999999999,"stat_sum":{"num_bytes":16646144,"num_objects":18,"num_object_clones":12,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":60,"num_read_kb":18326,"num_write":97,"num_write_kb":26448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,4,14,0,8,13,5],"acting":[12,4,14,0,8,13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.e","version":"109'171","reported_seq":554,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686575+0000","last_change":"2024-09-17T16:02:49.223276+0000","last_active":"2024-09-17T16:02:49.686575+0000","last_peered":"2024-09-17T16:02:49.686575+0000","last_clean":"2024-09-17T16:02:49.686575+0000","last_became_active":"2024-09-17T16:01:32.022338+0000","last_became_peered":"2024-09-17T16:01:32.022338+0000","last_unstale":"2024-09-17T16:02:49.686575+0000","last_undegraded":"2024-09-17T16:02:49.686575+0000","last_fullsized":"2024-09-17T16:02:49.686575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'171","last_scrub_stamp":"2024-09-17T16:02:49.223123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:49.223123+0000","objects_scrubbed":18,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:12.471242+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.013504957999999999,"stat_sum":{"num_bytes":20627456,"num_objects":18,"num_object_clones":14,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":38,"num_read_kb":11724,"num_write":105,"num_write_kb":17728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0,4,13,6,5],"acting":[2,11,0,4,13,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]},{"pgid":"3.f","version":"103'47","reported_seq":334,"reported_epoch":110,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:48.573532+0000","last_change":"2024-09-17T16:02:04.347090+0000","last_active":"2024-09-17T16:02:48.573532+0000","last_peered":"2024-09-17T16:02:48.573532+0000","last_clean":"2024-09-17T16:02:02.358329+0000","last_became_active":"2024-09-17T16:02:04.347090+0000","last_became_peered":"2024-09-17T16:02:04.347090+0000","last_unstale":"2024-09-17T16:02:48.573532+0000","last_undegraded":"2024-09-17T16:02:04.341882+0000","last_fullsized":"2024-09-17T16:02:04.341582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:30.823517+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015713103999999999,"stat_sum":{"num_bytes":2605056,"num_objects":9,"num_object_clones":6,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":21,"num_read_kb":1183,"num_write":40,"num_write_kb":2368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,14,11,2147483647,10,1],"acting":[0,2,14,11,2147483647,10,1],"avail_no_missing":["0(0)","1(6)","2(1)","10(5)","11(3)","14(2)"],"object_location_counts":[{"shards":"0(0),1(6),2(1),10(5),11(3),14(2)","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.10","version":"106'49","reported_seq":248,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.586496+0000","last_change":"2024-09-17T16:02:04.348295+0000","last_active":"2024-09-17T16:02:49.586496+0000","last_peered":"2024-09-17T16:02:49.586496+0000","last_clean":"2024-09-17T16:02:02.344595+0000","last_became_active":"2024-09-17T16:02:04.348295+0000","last_became_peered":"2024-09-17T16:02:04.348295+0000","last_unstale":"2024-09-17T16:02:49.586496+0000","last_undegraded":"2024-09-17T16:02:04.341600+0000","last_fullsized":"2024-09-17T16:02:04.341131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":5,"last_scrub":"56'31","last_scrub_stamp":"2024-09-17T16:01:58.728617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:58.728617+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:27.230200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.16505651299999999,"stat_sum":{"num_bytes":5537792,"num_objects":5,"num_object_clones":4,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":4176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,8,15,5,9,0],"acting":[2147483647,6,8,15,5,9,0],"avail_no_missing":["6(1)","0(6)","5(4)","8(2)","9(5)","15(3)"],"object_location_counts":[{"shards":"0(6),5(4),6(1),8(2),9(5),15(3)","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"76'18","reported_seq":220,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.568611+0000","last_change":"2024-09-17T16:02:04.349949+0000","last_active":"2024-09-17T16:02:49.568611+0000","last_peered":"2024-09-17T16:02:49.568611+0000","last_clean":"2024-09-17T16:02:02.345085+0000","last_became_active":"2024-09-17T16:02:04.349949+0000","last_became_peered":"2024-09-17T16:02:04.349949+0000","last_unstale":"2024-09-17T16:02:49.568611+0000","last_undegraded":"2024-09-17T16:02:04.342317+0000","last_fullsized":"2024-09-17T16:02:04.342004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'11","last_scrub_stamp":"2024-09-17T16:01:59.333371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:01:59.333371+0000","objects_scrubbed":2,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:23.674582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011690380000000001,"stat_sum":{"num_bytes":3538944,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":871,"num_write":4,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0,3,5,2147483647,1],"acting":[10,4,0,3,5,2147483647,1],"avail_no_missing":["10(0)","0(2)","1(6)","3(3)","4(1)","5(4)"],"object_location_counts":[{"shards":"0(2),1(6),3(3),4(1),5(4),10(0)","objects":3}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.12","version":"72'39","reported_seq":209,"reported_epoch":111,"state":"active+undersized+degraded","last_fresh":"2024-09-17T16:02:49.686497+0000","last_change":"2024-09-17T16:02:05.063706+0000","last_active":"2024-09-17T16:02:49.686497+0000","last_peered":"2024-09-17T16:02:49.686497+0000","last_clean":"2024-09-17T16:02:02.344908+0000","last_became_active":"2024-09-17T16:02:05.063706+0000","last_became_peered":"2024-09-17T16:02:05.063706+0000","last_unstale":"2024-09-17T16:02:49.686497+0000","last_undegraded":"2024-09-17T16:02:04.342739+0000","last_fullsized":"2024-09-17T16:02:04.342330+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":5,"last_scrub":"62'32","last_scrub_stamp":"2024-09-17T16:02:01.034730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:01.034730+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:12.812986+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0088005819999999995,"stat_sum":{"num_bytes":2129920,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,5,6,12,2,2147483647],"acting":[15,8,5,6,12,2,2147483647],"avail_no_missing":["15(0)","2(5)","5(2)","6(3)","8(1)","12(4)"],"object_location_counts":[{"shards":"2(5),5(2),6(3),8(1),12(4),15(0)","objects":3}],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"f","length":"1"}]},{"pgid":"3.13","version":"107'48","reported_seq":461,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-17T16:02:49.686450+0000","last_change":"2024-09-17T16:02:45.553328+0000","last_active":"2024-09-17T16:02:49.686450+0000","last_peered":"2024-09-17T16:02:49.686450+0000","last_clean":"2024-09-17T16:02:49.686450+0000","last_became_active":"2024-09-17T16:01:52.366019+0000","last_became_peered":"2024-09-17T16:01:52.366019+0000","last_unstale":"2024-09-17T16:02:49.686450+0000","last_undegraded":"2024-09-17T16:02:49.686450+0000","last_fullsized":"2024-09-17T16:02:49.686450+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":5,"last_scrub":"52'16","last_scrub_stamp":"2024-09-17T16:02:00.060603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:30.823517+0000","last_clean_scrub_stamp":"2024-09-17T16:02:00.060603+0000","objects_scrubbed":2,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:03:04.241755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0082142529999999995,"stat_sum":{"num_bytes":917504,"num_objects":3,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":531,"num_write":2,"num_write_kb":304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,8,1,0,2,4,12],"acting":[15,8,1,0,2,4,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"},{"start":"8","length":"3"},{"start":"d","length":"4"},{"start":"14","length":"2"},{"start":"17","length":"2"},{"start":"1a","length":"8"},{"start":"23","length":"2"},{"start":"27","length":"1"},{"start":"29","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":131317760,"num_objects":165,"num_object_clones":115,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":9,"num_read":307,"num_read_kb":124510,"num_write":578,"num_write_kb":114016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14319616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214564864,"data_stored":325214208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1570,"ondisk_log_size":1570,"up":182,"acting":182,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247831,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42232,"kb_used_data":11036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329608,"statfs":{"total":96636764160,"available":96593518592,"internally_reserved":0,"allocated":11300864,"data_stored":16195133,"data_compressed":277057,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280534,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39348,"kb_used_data":12320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332492,"statfs":{"total":96636764160,"available":96596471808,"internally_reserved":0,"allocated":12615680,"data_stored":16456903,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280535,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46760,"kb_used_data":15568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325080,"statfs":{"total":96636764160,"available":96588881920,"internally_reserved":0,"allocated":15941632,"data_stored":22742402,"data_compressed":273920,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313239,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49180,"kb_used_data":13832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322660,"statfs":{"total":96636764160,"available":96586403840,"internally_reserved":0,"allocated":14163968,"data_stored":18158604,"data_compressed":267646,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,4,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51792,"kb_used_data":20600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320048,"statfs":{"total":96636764160,"available":96583729152,"internally_reserved":0,"allocated":21094400,"data_stored":31071431,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55152,"kb_used_data":19800,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316688,"statfs":{"total":96636764160,"available":96580288512,"internally_reserved":0,"allocated":20275200,"data_stored":30334151,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280534,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43960,"kb_used_data":12764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327880,"statfs":{"total":96636764160,"available":96591749120,"internally_reserved":0,"allocated":13070336,"data_stored":19554023,"data_compressed":281282,"data_compressed_allocated":397312,"data_compressed_original":1232896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35772,"kb_used_data":8736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336068,"statfs":{"total":96636764160,"available":96600133632,"internally_reserved":0,"allocated":8945664,"data_stored":13581511,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52760,"kb_used_data":21564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319080,"statfs":{"total":96636764160,"available":96582737920,"internally_reserved":0,"allocated":22081536,"data_stored":33633666,"data_compressed":273920,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,6,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280534,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47788,"kb_used_data":16600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324052,"statfs":{"total":96636764160,"available":96587829248,"internally_reserved":0,"allocated":16998400,"data_stored":24595655,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49292,"kb_used_data":18104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322548,"statfs":{"total":96636764160,"available":96586289152,"internally_reserved":0,"allocated":18538496,"data_stored":29418882,"data_compressed":273920,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,5,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280524,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":5900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":6041600,"data_stored":8820876,"data_compressed":114770,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,4,5,6,8,9,10,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45672,"kb_used_data":14476,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326168,"statfs":{"total":96636764160,"available":96589996032,"internally_reserved":0,"allocated":14823424,"data_stored":21777626,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,5,6,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36044,"kb_used_data":9012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335796,"statfs":{"total":96636764160,"available":96599855104,"internally_reserved":0,"allocated":9228288,"data_stored":10806677,"data_compressed":273920,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,4,6,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37576,"kb_used_data":6384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334264,"statfs":{"total":96636764160,"available":96598286336,"internally_reserved":0,"allocated":6537216,"data_stored":8666311,"data_compressed":270783,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280533,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":20640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":21135360,"data_stored":32210663,"data_compressed":281282,"data_compressed_allocated":397312,"data_compressed_original":1232896,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19816448,"data_stored":30101504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12898304,"data_stored":17268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11735040,"data_stored":18055168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8261632,"data_stored":13639680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20791296,"data_stored":32653312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":15720448,"data_stored":23625728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":17248256,"data_stored":28438528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5378048,"data_stored":8372224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":14135296,"data_stored":21835776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":7933952,"data_stored":9826304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":5259264,"data_stored":7696384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":19800064,"data_stored":30711808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":10600448,"data_stored":16375808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":18997248,"data_stored":29364224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":14651392,"data_stored":21762048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":11337728,"data_stored":15486976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:02:53.331 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-17T16:02:53.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-17T16:02:53.366 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.366 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:53.366 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:53.366 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.371 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:53.369+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:53.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:53.465 INFO:tasks.ceph.osd.8.smithi006.stderr:2024-09-17T16:02:53.464+0000 7f5d6b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 33622) UID: 0 2024-09-17T16:02:53.477 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:53.477 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:53.478 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:53.566 INFO:tasks.ceph.osd.14.smithi169.stderr:2024-09-17T16:02:53.564+0000 7f7cdbe24640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 33031) UID: 0 2024-09-17T16:02:53.589 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 96.838065549999996, 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19491629299999999, 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.590 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748957+0000", 2024-09-17T16:02:53.591 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748958+0000", 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4300000000000001e-06 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748966+0000", 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.5870000000000004e-06 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.592 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748979+0000", 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2981e-05 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.748988+0000", 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.1929999999999999e-06 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749052+0000", 2024-09-17T16:02:53.593 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4618999999999999e-05 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.749064+0000", 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1437000000000001e-05 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.939947+0000", 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.19088328399999999 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940079+0000", 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013157500000000001 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:53.597 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.940245+0000", 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00016577900000000001 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943345+0000", 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0031004499999999998 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943806+0000", 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046112200000000001 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943843+0000", 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6578000000000001e-05 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.943873+0000", 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0258000000000001e-05 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4219.0:8 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [cmpxattr striper.excl (25) op 1 mode 1 in=12b,setxattr striper.excl in=12b,call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 96.638980223999994, 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83632086699999997, 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4219", 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/372584441", 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 8 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.600 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948042+0000", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948043+0000", 2024-09-17T16:02:53.601 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0999999999999997e-07 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948048+0000", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9540000000000003e-06 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948060+0000", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1761e-05 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948066+0000", 2024-09-17T16:02:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5350000000000001e-06 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948111+0000", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4783000000000001e-05 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948190+0000", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8893000000000005e-05 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 2", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948399+0000", 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000209292 2024-09-17T16:02:53.603 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.948922+0000", 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00052251800000000003 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784264+0000", 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83534246599999995 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.604 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784309+0000", 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4660000000000003e-05 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784363+0000", 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4095000000000003e-05 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_repop_reply(client.4219.0:8 1.0 e24/23 ondisk, result = 0)", 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:53.605 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 96.637858029, 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83521008600000002, 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949164+0000", 2024-09-17T16:02:53.606 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949165+0000", 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.99e-07 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:16.949169+0000", 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9339999999999999e-06 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.607 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784135+0000", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83496651200000005 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784151+0000", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.6079999999999999e-05 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784250+0000", 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.8222999999999998e-05 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.608 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784260+0000", 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0265000000000001e-05 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:17.784374+0000", 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000114373 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.609 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4296.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 94.827669815999997, 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.241196405, 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4296", 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.6:0/1428345084", 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 1 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.610 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759352+0000", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759356+0000", 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.484e-06 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.611 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759366+0000", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9089999999999995e-06 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759375+0000", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4960000000000002e-06 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759397+0000", 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.1824e-05 2024-09-17T16:02:53.612 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759444+0000", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.6539000000000003e-05 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for peered", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.759449+0000", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8139999999999997e-06 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998167+0000", 2024-09-17T16:02:53.613 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.23871730099999999 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998242+0000", 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4892000000000001e-05 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for subops from 0", 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998303+0000", 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0931999999999999e-05 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.614 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op_commit", 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:18.998708+0000", 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00040511399999999998 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_commit_rec", 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000499+0000", 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0017911909999999999 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.615 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000526+0000", 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6798999999999999e-05 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:19.000549+0000", 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3110000000000001e-05 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.616 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:34 3.2s0 3:41aa4422:::smithi18136759-12:head [append 0~196608,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.730407311999997, 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015312129000000001, 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 34 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.617 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856615+0000", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856616+0000", 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4899999999999999e-06 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.618 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856732+0000", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011513 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856745+0000", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3677e-05 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856753+0000", 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.6930000000000007e-06 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.619 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856824+0000", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1291999999999994e-05 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856962+0000", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00013730799999999999 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.857708+0000", 2024-09-17T16:02:53.620 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00074624700000000001 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861225+0000", 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0035171249999999999 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871848+0000", 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010623323 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.621 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.871927+0000", 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8844000000000006e-05 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:35 3.2s0 3:41aa4422:::smithi18136759-12:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.730251772000003, 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015326866, 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.622 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 35 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.623 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856770+0000", 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856771+0000", 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4000000000000001e-07 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856775+0000", 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.33e-06 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.624 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856782+0000", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0800000000000002e-06 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856786+0000", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5759999999999999e-06 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858308+0000", 2024-09-17T16:02:53.625 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015218009999999999 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.858393+0000", 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5502999999999998e-05 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859025+0000", 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000632009 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.626 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.861267+0000", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0022417639999999998 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872026+0000", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010758772999999999 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872097+0000", 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1390000000000006e-05 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.627 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:36 3.2s0 3:41aa4422:::smithi18136759-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.730166428999993, 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017199677, 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.628 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 36 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:53.629 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856856+0000", 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3300000000000002e-07 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856861+0000", 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0759999999999996e-06 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.630 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856869+0000", 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6319999999999997e-06 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.856876+0000", 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.5989999999999998e-06 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859529+0000", 2024-09-17T16:02:53.631 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026531060000000001 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.859628+0000", 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9104000000000005e-05 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872415+0000", 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012787056 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.632 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.872500+0000", 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4989999999999998e-05 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874055+0000", 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015554810000000001 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=11, reqid=client.4560.0:50, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:53.633 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.716082764000006, 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016547899000000001, 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870939+0000", 2024-09-17T16:02:53.634 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870940+0000", 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1600000000000001e-07 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.870953+0000", 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2798e-05 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.635 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881713+0000", 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010760202 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881721+0000", 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8949999999999998e-06 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881765+0000", 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3778e-05 2024-09-17T16:02:53.636 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.881775+0000", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0135999999999999e-05 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887451+0000", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0056764090000000003 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.887487+0000", 2024-09-17T16:02:53.637 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.6065000000000001e-05 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:81 3.2s0 3:4c2fa52d:::smithi18136759-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.692978964999995, 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015999928, 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.638 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 81 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.639 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894043+0000", 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894044+0000", 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8099999999999999e-07 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894045+0000", 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.065e-06 2024-09-17T16:02:53.640 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894048+0000", 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.489e-06 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.894051+0000", 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9799999999999998e-06 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.641 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896400+0000", 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0023486919999999999 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.896473+0000", 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3380000000000006e-05 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908531+0000", 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012058016 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.642 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.908615+0000", 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3968000000000007e-05 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.910043+0000", 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0014280569999999999 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.643 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.1s3 35/32 ECSubWrite(tid=34, reqid=client.4560.0:104, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.661623113999994, 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021043124999999999, 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:53.644 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925399+0000", 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925400+0000", 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7599999999999997e-07 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.645 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.925410+0000", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0345e-05 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944975+0000", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.019565013999999999 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944985+0000", 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0257e-05 2024-09-17T16:02:53.646 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945787+0000", 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00080139199999999995 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945797+0000", 2024-09-17T16:02:53.647 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9590000000000007e-06 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946410+0000", 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00061359000000000001 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.946442+0000", 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1891999999999999e-05 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.648 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.7s4 35/32 ECSubWrite(tid=42, reqid=client.4560.0:128, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 79.631265089999999, 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.022376667999999999, 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.649 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955757+0000", 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955758+0000", 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0699999999999997e-07 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.650 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.955765+0000", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.345e-06 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977438+0000", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.021672416999999999 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977447+0000", 2024-09-17T16:02:53.651 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7210000000000006e-06 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977511+0000", 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4066000000000006e-05 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.977519+0000", 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8329999999999994e-06 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.652 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978105+0000", 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058612099999999999 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.978134+0000", 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8558e-05 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.653 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:268 3.2s0 3:406bd9c9:::smithi18136759-23:head [append 0~720896] snapc a=[a,9,8,7,6,5,3,2] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.663698585999995, 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.016657371000000001, 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 268 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.654 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923324+0000", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923326+0000", 2024-09-17T16:02:53.655 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0999999999999998e-06 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923806+0000", 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00048034 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923821+0000", 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5180999999999999e-05 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.656 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923831+0000", 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4450000000000002e-06 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923891+0000", 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.9858999999999997e-05 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.923971+0000", 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0493000000000003e-05 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.657 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.924982+0000", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001010543 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929374+0000", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0043921699999999999 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939921+0000", 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.010546754 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.658 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.939981+0000", 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0486000000000003e-05 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:271 3.2s0 3:406bd9c9:::smithi18136759-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.661062517000005, 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015374301, 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.659 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 271 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925960+0000", 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.660 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925961+0000", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8800000000000002e-07 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925964+0000", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5319999999999998e-06 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925976+0000", 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1795e-05 2024-09-17T16:02:53.661 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.925983+0000", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.289e-06 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929232+0000", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0032491070000000002 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.929352+0000", 2024-09-17T16:02:53.662 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000119639 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940555+0000", 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011203489000000001 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.940634+0000", 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8165999999999997e-05 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.663 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:43.941334+0000", 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000700496 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:392 3.2s0 3:465e1f11:::smithi18136759-5:head [append 0~589824] snapc 13=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 55.244545887000001, 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.018219264999999998, 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.664 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 392 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342476+0000", 2024-09-17T16:02:53.665 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.342479+0000", 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.481e-06 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343310+0000", 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00083096699999999995 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.666 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343328+0000", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8059000000000001e-05 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343340+0000", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1664e-05 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343445+0000", 2024-09-17T16:02:53.667 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010582900000000001 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.343453+0000", 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7999999999999999e-06 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348803+0000", 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053503420000000001 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.668 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.348975+0000", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017183899999999999 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.350509+0000", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0015333320000000001 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356400+0000", 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0058917320000000002 2024-09-17T16:02:53.669 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360612+0000", 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042120680000000002 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.360696+0000", 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3152000000000006e-05 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.670 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:396 3.2s0 3:465e1f11:::smithi18136759-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 55.239284480999999, 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.015698437999999999, 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 396 2024-09-17T16:02:53.671 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.672 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347738+0000", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.8699999999999995e-07 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347741+0000", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.6690000000000002e-06 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347747+0000", 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.455e-06 2024-09-17T16:02:53.673 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.347752+0000", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0459999999999996e-06 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356238+0000", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084855650000000005 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.356325+0000", 2024-09-17T16:02:53.674 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7039999999999993e-05 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361673+0000", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0053483849999999998 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.361762+0000", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8825000000000004e-05 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:58.363436+0000", 2024-09-17T16:02:53.675 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001673966 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:484 3.18s0 3:19aefe3f:::smithi18136759-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.236064380999998, 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.017624631000000002, 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.676 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 484 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.677 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350958+0000", 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8400000000000004e-07 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.350961+0000", 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0000000000000001e-06 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.678 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352262+0000", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001300708 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.352278+0000", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5848000000000001e-05 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354256+0000", 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019783330000000001 2024-09-17T16:02:53.679 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.354327+0000", 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0839000000000004e-05 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363568+0000", 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0092410559999999992 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.680 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.363668+0000", 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.9382000000000005e-05 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368582+0000", 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0049148810000000003 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:488 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [setxattr _header (57) in=64b] snapc 17=[17,16,14,10,e,d,c,9,8,6] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:53.681 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.222902445999999, 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83388352600000004, 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 488 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.682 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364120+0000", 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9900000000000001e-07 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.683 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364123+0000", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9220000000000001e-06 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364131+0000", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3010000000000001e-06 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364137+0000", 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4679999999999997e-06 2024-09-17T16:02:53.684 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366664+0000", 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0025266949999999998 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.366744+0000", 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0739000000000001e-05 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:53.685 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367168+0000", 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042360999999999999 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368066+0000", 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00089781599999999996 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197852+0000", 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82978574900000002 2024-09-17T16:02:53.686 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198003+0000", 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00015172699999999999 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:489 3.18s0 3:1d4c1f5f:::smithi18136759-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-17T16:02:53.687 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.222876225999997, 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83494334199999998, 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:53.688 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 489 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.689 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364146+0000", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364147+0000", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9999999999999998e-07 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364149+0000", 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2359999999999999e-06 2024-09-17T16:02:53.690 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364156+0000", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4630000000000004e-06 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.364163+0000", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9190000000000002e-06 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368609+0000", 2024-09-17T16:02:53.691 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044454869999999997 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.368658+0000", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.9438999999999999e-05 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198316+0000", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965781999999999 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198436+0000", 2024-09-17T16:02:53.692 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012003299999999999 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.199089+0000", 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000653445 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.18s0 69/53 ECSubWriteReply(tid=224, last_complete=69'42, committed=1, applied=1))", 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 46.219097509000001, 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.83034511799999999, 2024-09-17T16:02:53.693 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.694 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367925+0000", 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.13e-07 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:07.367929+0000", 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3569999999999998e-06 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.695 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197585+0000", 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.82965586700000005 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197599+0000", 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4369000000000001e-05 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.197711+0000", 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00011220199999999999 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.696 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:08.198270+0000", 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00055871000000000002 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpRead(3.19s1 86/53 ECSubRead(tid=180, to_read={3:9b2471fe:::smithi18136759-32:17=114688,16384,0}, subchunks={3:9b2471fe:::smithi18136759-32:17=[0,1]}, attrs_to_read=))", 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 29.123996969, 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.43055667600000003, 2024-09-17T16:02:53.697 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463025+0000", 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.698 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463026+0000", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.09e-07 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.463035+0000", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.2639999999999997e-06 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893268+0000", 2024-09-17T16:02:53.699 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.430232958 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893300+0000", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2008999999999998e-05 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893479+0000", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00017834700000000001 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:24.893582+0000", 2024-09-17T16:02:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010328900000000001 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:53.701 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:53.703 INFO:tasks.ceph.osd.0.smithi006.stderr:2024-09-17T16:02:53.700+0000 7f45eba31640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33617) UID: 0 2024-09-17T16:02:53.803 INFO:tasks.ceph.osd.9.smithi050.stderr:2024-09-17T16:02:53.801+0000 7f12e8239640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 33112) UID: 0 2024-09-17T16:02:53.806 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.806 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:53.806 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:02:53.806 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.815 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:53.904 INFO:tasks.ceph.osd.12.smithi006.stderr:2024-09-17T16:02:53.902+0000 7f91bcfe6640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 33625) UID: 0 2024-09-17T16:02:53.918 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:53.918 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:02:53.919 INFO:teuthology.orchestra.run.smithi181.stdout: "complaint_time": 30, 2024-09-17T16:02:53.919 INFO:teuthology.orchestra.run.smithi181.stdout: "num_blocked_ops": 0 2024-09-17T16:02:53.919 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:53.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:54.008 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-17T16:02:54.004+0000 7f9a0281d640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33261) UID: 0 2024-09-17T16:02:54.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "size": 20, 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 600, 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=7, reqid=client.4560.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:54.032 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.161826417, 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066769949999999998, 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.033 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867463+0000", 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867464+0000", 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5900000000000005e-07 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867602+0000", 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000138226 2024-09-17T16:02:54.034 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867633+0000", 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0892999999999997e-05 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867642+0000", 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4290000000000001e-06 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867699+0000", 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7055000000000002e-05 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867707+0000", 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3129999999999999e-06 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.036 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874101+0000", 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063937710000000003 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874140+0000", 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9348999999999999e-05 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.037 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.6s2 35/32 ECSubWrite(tid=9, reqid=client.4560.0:47, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.161436705, 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0063169489999999997, 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.038 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867853+0000", 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867854+0000", 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.99e-07 2024-09-17T16:02:54.039 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867869+0000", 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5133000000000001e-05 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867900+0000", 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0864000000000001e-05 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.040 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.867907+0000", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.8150000000000006e-06 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868182+0000", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00027485999999999998 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.868190+0000", 2024-09-17T16:02:54.041 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4309999999999997e-06 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874148+0000", 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059580780000000003 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.874170+0000", 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2169000000000001e-05 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.042 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.8s3 35/32 ECSubWrite(tid=125, reqid=client.4560.0:59, at_version=35'4, trim_to=0'0, roll_forward_to=34'2))", 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.152016295999999, 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077692500000000001, 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.043 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877273+0000", 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.044 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877274+0000", 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.4499999999999996e-07 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877292+0000", 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8134999999999998e-05 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.045 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877325+0000", 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.2892999999999998e-05 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877334+0000", 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7739999999999993e-06 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877406+0000", 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2008000000000006e-05 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.046 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.877415+0000", 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5179999999999993e-06 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.884998+0000", 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0075826280000000001 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.047 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.885043+0000", 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4549000000000003e-05 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:109 3.3s0 3:c9c73df9:::smithi18136759-37:head [append 0~606208,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.097330771000003, 2024-09-17T16:02:54.048 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061210880000000002, 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.049 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 109 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931959+0000", 2024-09-17T16:02:54.050 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.931960+0000", 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.22e-06 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932306+0000", 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00034552299999999998 2024-09-17T16:02:54.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932316+0000", 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0142e-05 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932324+0000", 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1790000000000008e-06 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.052 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932396+0000", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2512e-05 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932547+0000", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000151008 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.933463+0000", 2024-09-17T16:02:54.053 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00091551299999999996 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935373+0000", 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001910166 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937996+0000", 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0026225189999999998 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.054 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938080+0000", 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4306000000000002e-05 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:110 3.3s0 3:c9c73df9:::smithi18136759-37:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.096948372, 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.006799411, 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.055 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 110 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.056 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932341+0000", 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932342+0000", 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.1600000000000002e-07 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.057 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932344+0000", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.187e-06 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932349+0000", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7160000000000002e-06 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932352+0000", 2024-09-17T16:02:54.058 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0790000000000001e-06 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934491+0000", 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021397529999999999 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.934586+0000", 2024-09-17T16:02:54.059 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.4180999999999998e-05 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935015+0000", 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042948699999999999 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937773+0000", 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00275793 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.060 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939052+0000", 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012792470000000001 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939141+0000", 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8415e-05 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.061 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:111 3.3s0 3:c9c73df9:::smithi18136759-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.096912579000005, 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0084761379999999994, 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.062 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 111 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.063 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932377+0000", 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.7700000000000001e-07 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932379+0000", 2024-09-17T16:02:54.064 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.824e-06 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932384+0000", 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.7840000000000003e-06 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932392+0000", 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0709999999999995e-06 2024-09-17T16:02:54.065 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935233+0000", 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002841297 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935311+0000", 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7315000000000001e-05 2024-09-17T16:02:54.066 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939191+0000", 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038802670000000002 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939273+0000", 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1680000000000004e-05 2024-09-17T16:02:54.067 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940853+0000", 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001580623 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:112 3.3s0 3:c1daf1ed:::smithi18136759-38:head [append 0~622592,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.068 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.096884414000002, 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0076371060000000003, 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 112 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.069 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932405+0000", 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.070 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932406+0000", 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3999999999999997e-07 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932660+0000", 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00025410199999999997 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.071 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932666+0000", 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0680000000000002e-06 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932673+0000", 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.1749999999999999e-06 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.072 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935440+0000", 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027674890000000001 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.935561+0000", 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000121032 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.073 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936338+0000", 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00077618200000000004 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937849+0000", 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001511517 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.939978+0000", 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002128352 2024-09-17T16:02:54.074 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.940042+0000", 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4849000000000002e-05 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:113 3.3s0 3:c1daf1ed:::smithi18136759-38:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:54.075 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.096610894999998, 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.011038628, 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 113 2024-09-17T16:02:54.076 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.077 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932679+0000", 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3599999999999999e-07 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932681+0000", 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.646e-06 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.078 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932684+0000", 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.242e-06 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932687+0000", 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.666e-06 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.936959+0000", 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0042721970000000001 2024-09-17T16:02:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937067+0000", 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000108245 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.937523+0000", 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000455909 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.080 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938194+0000", 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00067141300000000004 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943613+0000", 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0054180859999999999 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.943717+0000", 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000104888 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.081 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:114 3.3s0 3:c1daf1ed:::smithi18136759-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.096599839000007, 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.012592028, 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.082 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 114 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.083 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932690+0000", 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.3999999999999998e-07 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932691+0000", 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4360000000000001e-06 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932696+0000", 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.0740000000000003e-06 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.932698+0000", 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8660000000000002e-06 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.085 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938448+0000", 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057490739999999999 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.938551+0000", 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00010375800000000001 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944111+0000", 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0055593070000000003 2024-09-17T16:02:54.086 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.944207+0000", 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5970000000000004e-05 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.945282+0000", 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0010753029999999999 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.087 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:136 3.3s0 3:c557bac7:::smithi18136759-46:head [append 0~770048,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.063844685999996, 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0066308310000000002, 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.088 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 136 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.089 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965445+0000", 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965447+0000", 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0839999999999999e-06 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.090 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965888+0000", 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00044097500000000003 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965907+0000", 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.861e-05 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.091 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965916+0000", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0059999999999998e-06 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965967+0000", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.1768000000000003e-05 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966063+0000", 2024-09-17T16:02:54.092 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.5257e-05 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.966714+0000", 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00065150699999999995 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967960+0000", 2024-09-17T16:02:54.093 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0012458980000000001 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972031+0000", 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0040714210000000004 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972076+0000", 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4304999999999997e-05 2024-09-17T16:02:54.094 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:137 3.3s0 3:c557bac7:::smithi18136759-46:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.063355635999997, 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062731460000000003, 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.095 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 137 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.096 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965934+0000", 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965935+0000", 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.6899999999999997e-07 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.097 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965940+0000", 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.0309999999999998e-06 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965948+0000", 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0679999999999997e-06 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965954+0000", 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.1519999999999997e-06 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.098 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967258+0000", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00130415 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967322+0000", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.3922000000000002e-05 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967744+0000", 2024-09-17T16:02:54.099 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00042239900000000001 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968311+0000", 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00056638000000000001 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972135+0000", 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0038243330000000001 2024-09-17T16:02:54.100 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972207+0000", 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2041999999999999e-05 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:138 3.3s0 3:c557bac7:::smithi18136759-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-17T16:02:54.101 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 80.063314786000007, 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071879300000000004, 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "started", 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 138 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.102 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.103 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965975+0000", 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.8699999999999995e-07 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965978+0000", 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.8119999999999999e-06 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.104 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965987+0000", 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.4149999999999999e-06 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.965998+0000", 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1813e-05 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.105 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.967989+0000", 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0019901369999999999 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "waiting for rw locks", 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.968074+0000", 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.5296000000000005e-05 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.106 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972477+0000", 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0044034690000000001 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.972555+0000", 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7403000000000004e-05 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:33.973163+0000", 2024-09-17T16:02:54.107 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000607998 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.4s4 46/32 ECSubWrite(tid=15, reqid=osd.1.0:15, at_version=46'14, trim_to=0'0, roll_forward_to=45'12))", 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.115595189999993, 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0073441670000000004, 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.108 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913694+0000", 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.109 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913695+0000", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.1899999999999998e-07 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.913725+0000", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9743999999999998e-05 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919758+0000", 2024-09-17T16:02:54.110 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00603318 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919766+0000", 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3610000000000001e-06 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919835+0000", 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.9336000000000002e-05 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.111 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.919844+0000", 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9970000000000004e-06 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921001+0000", 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00115653 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.112 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.921039+0000", 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.7700000000000002e-05 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 46/32 ECSubWriteReply(tid=17, last_complete=46'12, committed=1, applied=1))", 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 69.105608613000001, 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.330502611, 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.113 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923681+0000", 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.114 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923682+0000", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.1999999999999998e-07 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:44.923686+0000", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.2379999999999999e-06 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253962+0000", 2024-09-17T16:02:54.115 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.33027621400000001 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.253975+0000", 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3159000000000001e-05 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254119+0000", 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014402100000000001 2024-09-17T16:02:54.116 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:45.254184+0000", 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4158999999999993e-05 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:346 3.3s0 3:c1daf1ed:::smithi18136759-38:head [delete] snapc 10=[10,f,e,d,c,a,9,8,6,5,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e54)", 2024-09-17T16:02:54.117 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 61.670519792, 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069854690000000002, 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 346 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.118 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358770+0000", 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.119 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358772+0000", 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.0810000000000001e-06 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358779+0000", 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.2849999999999997e-06 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358790+0000", 2024-09-17T16:02:54.120 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.1124999999999999e-05 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358798+0000", 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9759999999999995e-06 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358838+0000", 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.9730000000000001e-05 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.121 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.358966+0000", 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012749199999999999 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.359434+0000", 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00046890000000000001 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.122 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.362289+0000", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0028544030000000001 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365699+0000", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034100210000000001 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:01:52.365755+0000", 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6456000000000003e-05 2024-09-17T16:02:54.123 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.10s3 64/57 ECSubWrite(tid=6, reqid=osd.7.0:6, at_version=64'35, trim_to=0'0, roll_forward_to=62'33))", 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.683148352000003, 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0081444710000000003, 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.124 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.125 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346141+0000", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.8199999999999999e-07 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346159+0000", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7445e-05 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346307+0000", 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00014812199999999999 2024-09-17T16:02:54.126 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346314+0000", 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.477e-06 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346436+0000", 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00012286999999999999 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.127 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.346445+0000", 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.833e-06 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354242+0000", 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0077967399999999999 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354286+0000", 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3801999999999998e-05 2024-09-17T16:02:54.128 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.16s2 64/53 ECSubWrite(tid=133, reqid=osd.13.0:132, at_version=0'0, trim_to=0'0, roll_forward_to=64'20))", 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 51.680816530999998, 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0059176979999999999, 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.129 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348473+0000", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348474+0000", 2024-09-17T16:02:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.5799999999999999e-07 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348485+0000", 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0892e-05 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348510+0000", 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.5287999999999999e-05 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.131 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348517+0000", 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.267e-06 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348561+0000", 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.4360000000000002e-05 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.348568+0000", 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.7270000000000001e-06 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.132 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354355+0000", 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0057869480000000001 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:02.354391+0000", 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5658000000000003e-05 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.133 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.14s4 81/53 ECSubWrite(tid=38, reqid=osd.1.0:38, at_version=81'30, trim_to=0'0, roll_forward_to=79'28))", 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.601704523000002, 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0071372670000000001, 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.134 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.135 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427585+0000", 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.48e-07 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427607+0000", 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.2113e-05 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427687+0000", 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9518000000000007e-05 2024-09-17T16:02:54.136 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427696+0000", 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.6680000000000007e-06 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427760+0000", 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.4349000000000003e-05 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.137 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.427770+0000", 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0236e-05 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.434677+0000", 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0069065589999999996 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.434722+0000", 2024-09-17T16:02:54.138 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5575999999999999e-05 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(3.19s3 81/53 ECSubWrite(tid=157, reqid=client.4560.0:672, at_version=81'43, trim_to=0'0, roll_forward_to=81'39))", 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 34.576809967999999, 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0061579099999999999, 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.139 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452480+0000", 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.140 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452481+0000", 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.7000000000000003e-07 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452813+0000", 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00033278099999999998 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.141 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452847+0000", 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.3637000000000003e-05 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452853+0000", 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.7309999999999998e-06 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.142 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452933+0000", 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0384999999999997e-05 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.452944+0000", 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0900000000000001e-05 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.458602+0000", 2024-09-17T16:02:54.143 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.005657825 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:19.458638+0000", 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5781000000000002e-05 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "osd_op(client.4560.0:979 3.12s0 3:4c2fa52d:::smithi18136759-27:head [rmxattr NikdcN_ilNkWgOQgZqrYPT\\b[drPRq\\niPPQdbVU in=40b,rmxattr SYo`knsRbgPQPPbXW[[ZsTehoXbSPY_WQl]_URc` in=40b,rmxattr \\PYbW]PkbTb^sYhsihYgqj]pROo]\\NWqNhT_lWgn in=40b,rmxattr aRNZ_mZpadm^l[rZntn`UX^RN\\cZ_TT]rRTjibjc in=40b,rmxattr c[X]j^UsOa`q]sYrSaa]raqNm]iSd^X]OmelObeN in=40b,rmxattr nNhpa\\e^VfUjte_Pc[kSor^[gpr[eaflXRs]SWZr in=40b] snapc 30=[30,2f,2e,2d,2c,2b,2a,28,26,25,22,19,16,c,6] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-17T16:02:54.144 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-17T16:02:52.585216+0000", 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 1.444073309, 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0064300939999999999, 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "client_info": { 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "client": "client.4560", 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "client_addr": "172.21.15.181:0/441480445", 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: "tid": 979 2024-09-17T16:02:54.145 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585216+0000", 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585216+0000", 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-17T16:02:54.146 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585218+0000", 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.7999999999999999e-06 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585226+0000", 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9330000000000001e-06 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.147 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585242+0000", 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5679e-05 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585258+0000", 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.5956000000000001e-05 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.148 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585318+0000", 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 6.0421999999999998e-05 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "op must wait for map", 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.585327+0000", 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.3850000000000002e-06 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.149 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.588060+0000", 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0027333050000000001 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "started", 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.588157+0000", 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7668000000000001e-05 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-17T16:02:54.150 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.588875+0000", 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00071720200000000001 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.590411+0000", 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.001536653 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "commit_sent", 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.591511+0000", 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0011000789999999999 2024-09-17T16:02:54.151 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "done", 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-17T16:02:52.591646+0000", 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000135012 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-17T16:02:54.152 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:02:54.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-09-17T16:02:54.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-17T16:02:54.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-17T16:02:54.155 INFO:tasks.ceph.osd.10.smithi169.stderr:2024-09-17T16:02:54.152+0000 7f81fbd9a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 33027) UID: 0 2024-09-17T16:02:54.169 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-09-17T16:02:54.169 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-17T16:02:54.190 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-09-17T16:02:54.191 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-17T16:02:54.199 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [7] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 11, 15] 2024-09-17T16:02:54.199 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-17T16:02:54.199 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-17T16:02:54.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-09-17T16:02:54.212 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-09-17T16:02:54.212 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-17T16:02:54.233 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-09-17T16:02:54.233 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-17T16:02:54.254 INFO:tasks.ceph.osd.13.smithi050.stderr:2024-09-17T16:02:54.252+0000 7f8919546640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 33115) UID: 0 2024-09-17T16:02:54.255 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-09-17T16:02:54.255 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-17T16:02:54.265 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.266 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-17T16:02:54.276 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-09-17T16:02:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-17T16:02:54.277 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-17T16:02:54.298 INFO:teuthology.orchestra.run.smithi006.stdout:osd.8: {} 2024-09-17T16:02:54.299 INFO:teuthology.orchestra.run.smithi006.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-17T16:02:54.322 INFO:teuthology.orchestra.run.smithi006.stdout:osd.9: {} 2024-09-17T16:02:54.322 INFO:teuthology.orchestra.run.smithi006.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-17T16:02:54.356 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-17T16:02:54.354+0000 7fa4d080e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33107) UID: 0 2024-09-17T16:02:54.359 INFO:teuthology.orchestra.run.smithi006.stdout:osd.10: {} 2024-09-17T16:02:54.359 INFO:teuthology.orchestra.run.smithi006.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-17T16:02:54.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.11: {} 2024-09-17T16:02:54.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-17T16:02:54.386 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.386 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-17T16:02:54.401 INFO:teuthology.orchestra.run.smithi006.stdout:osd.12: {} 2024-09-17T16:02:54.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-17T16:02:54.425 INFO:teuthology.orchestra.run.smithi006.stdout:osd.13: {} 2024-09-17T16:02:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-17T16:02:54.456 INFO:tasks.ceph.osd.15.smithi181.stderr:2024-09-17T16:02:54.455+0000 7f84a7e7c640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 33270) UID: 0 2024-09-17T16:02:54.463 INFO:teuthology.orchestra.run.smithi006.stdout:osd.14: {} 2024-09-17T16:02:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-17T16:02:54.498 INFO:teuthology.orchestra.run.smithi006.stdout:osd.15: {} 2024-09-17T16:02:54.498 INFO:teuthology.orchestra.run.smithi006.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-17T16:02:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-17T16:02:54.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-17T16:02:54.530 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.530 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.535 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-17T16:02:54.534+0000 7f62d6495640 -1 mon.a@0(leader).osd e114 definitely_dead 0 2024-09-17T16:02:54.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-17T16:02:54.558 INFO:tasks.ceph.osd.2.smithi169.stderr:2024-09-17T16:02:54.556+0000 7f6198231640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33024) UID: 0 2024-09-17T16:02:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' removed 2024-09-17T16:02:54.585 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.7. 2024-09-17T16:02:54.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-17T16:02:54.608 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-17T16:02:54.621 INFO:tasks.thrashosds:joining thrashosds 2024-09-17T16:02:54.694 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.694 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-17T16:02:54.811 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.811 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-17T16:02:54.927 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:54.927 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:54.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-17T16:02:55.044 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.044 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-17T16:02:55.159 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.159 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-17T16:02:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-17T16:02:55.390 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.390 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-17T16:02:55.505 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.505 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-17T16:02:55.596 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-09-17T16:02:55.626 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.626 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-17T16:02:55.754 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.754 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-17T16:02:55.873 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.873 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-17T16:02:55.989 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:55.989 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:55.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-17T16:02:56.105 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.105 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-17T16:02:56.223 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.224 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-17T16:02:56.339 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.340 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-17T16:02:56.455 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.455 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-17T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.570 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-17T16:02:56.687 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.687 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-17T16:02:56.804 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.804 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-17T16:02:56.919 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:56.919 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:56.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-17T16:02:57.034 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.034 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-17T16:02:57.159 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.159 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-17T16:02:57.273 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.273 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.282 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-17T16:02:57.385 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.385 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.394 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-17T16:02:57.497 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.497 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.505 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-17T16:02:57.609 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.609 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-17T16:02:57.618 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-17T16:02:57.721 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.721 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-17T16:02:57.834 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.834 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-17T16:02:57.946 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:57.946 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:57.954 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-17T16:02:58.058 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.058 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.066 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-17T16:02:58.170 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.170 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.179 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-17T16:02:58.282 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.282 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-17T16:02:58.393 INFO:teuthology.orchestra.run.smithi169.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.393 INFO:teuthology.orchestra.run.smithi169.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-17T16:02:58.517 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.517 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-17T16:02:58.603 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-09-17T16:02:58.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-17T16:02:58.629 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.629 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-17T16:02:58.741 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.741 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-17T16:02:58.852 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.852 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-17T16:02:58.962 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:58.962 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:58.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-17T16:02:59.073 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:59.074 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:59.083 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-17T16:02:59.185 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:59.185 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:59.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-17T16:02:59.296 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:59.296 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:59.304 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-17T16:02:59.407 INFO:teuthology.orchestra.run.smithi181.stdout:ERROR: (22) Invalid argument 2024-09-17T16:02:59.407 INFO:teuthology.orchestra.run.smithi181.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-17T16:02:59.608 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-09-17T16:02:59.622 INFO:tasks.thrashosds.thrasher:reviving osd 2024-09-17T16:02:59.622 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-17T16:02:59.622 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-17T16:02:59.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-17T16:02:59.625 INFO:tasks.ceph.osd.7:Started 2024-09-17T16:02:59.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-17T16:02:59.724 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-17T16:02:59.725 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-17T16:02:59.725 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-17T16:02:59.725 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-17T16:02:59.725 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-17T16:02:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:02:59.727 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-17T16:03:00.793 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:03:00.790+0000 7fbf5e5cc640 -1 Falling back to public interface 2024-09-17T16:03:04.562 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:03:04.559+0000 7fbf5e5cc640 -1 osd.7 64 log_to_monitors true 2024-09-17T16:03:04.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-17T16:03:04.844 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs debug_inject_read_zeros 2024-09-17T16:03:04.845 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs files list 2024-09-17T16:03:04.845 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs stats 2024-09-17T16:03:04.845 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator dump block 2024-09-17T16:03:04.845 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator fragmentation block 2024-09-17T16:03:04.845 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-17T16:03:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-17T16:03:04.847 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-17T16:03:05.203 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T16:03:05.199+0000 7fbf5237a640 -1 osd.7 119 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T16:03:09.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-17T16:03:09.965 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-17T16:03:09.965 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-17T16:03:09.965 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-17T16:03:09.965 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-17T16:03:09.974 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-17T16:03:09.974 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-17T16:03:09.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-09-17T16:03:10.301 INFO:tasks.ceph.mon.a.smithi006.stderr:2024-09-17T16:03:10.299+0000 7f62d6495640 -1 mon.a@0(leader).osd e121 definitely_dead 0 2024-09-17T16:03:11.192 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.7. 2024-09-17T16:03:11.209 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-17T16:03:11.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-17T16:03:11.341 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-09-17T16:03:11.342 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:03:11.356 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-09-17T16:03:11.356 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:03:11.371 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-09-17T16:03:11.371 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-17T16:03:11.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-17T16:03:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-09-17T16:03:12.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-17T16:03:13.203 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-17T16:03:13.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-17T16:03:14.204 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-09-17T16:03:14.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-17T16:03:15.210 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-17T16:03:15.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-17T16:03:16.220 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.8 to 1 (10000) 2024-09-17T16:03:16.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-17T16:03:17.228 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-17T16:03:17.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-17T16:03:18.237 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.12 to 1 (10000) 2024-09-17T16:03:18.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-17T16:03:19.243 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-17T16:03:19.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-17T16:03:20.248 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-09-17T16:03:20.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-17T16:03:21.254 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-17T16:03:21.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-17T16:03:22.261 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.5 to 1 (10000) 2024-09-17T16:03:22.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-17T16:03:23.266 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-17T16:03:23.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-17T16:03:24.273 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.9 to 1 (10000) 2024-09-17T16:03:24.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-17T16:03:25.286 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-17T16:03:25.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-17T16:03:26.294 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.13 to 1 (10000) 2024-09-17T16:03:26.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-17T16:03:27.300 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-17T16:03:27.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-17T16:03:28.307 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-09-17T16:03:28.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-17T16:03:29.317 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-17T16:03:29.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-17T16:03:30.327 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.6 to 1 (10000) 2024-09-17T16:03:30.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-17T16:03:31.333 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-17T16:03:31.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-17T16:03:32.339 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.10 to 1 (10000) 2024-09-17T16:03:32.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-17T16:03:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-17T16:03:33.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-17T16:03:34.352 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.14 to 1 (10000) 2024-09-17T16:03:34.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-17T16:03:35.358 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-17T16:03:35.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-17T16:03:36.363 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.3 to 1 (10000) 2024-09-17T16:03:36.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-17T16:03:37.381 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-17T16:03:37.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-17T16:03:38.386 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.11 to 1 (10000) 2024-09-17T16:03:38.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-17T16:03:39.388 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-17T16:03:39.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-17T16:03:40.393 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.15 to 1 (10000) 2024-09-17T16:03:40.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-17T16:03:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-17T16:03:41.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-17T16:03:42.403 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-09-17T16:03:42.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-17T16:03:43.410 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-17T16:03:43.427 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-17T16:03:43.428 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-17T16:03:43.428 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-17T16:03:43.428 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-17T16:03:43.428 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T16:03:43.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:03:43.743 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:43.743 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":154,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:03:43.406922+0000","last_up_change":"2024-09-17T16:03:06.175857+0000","last_in_change":"2024-09-17T16:03:11.183940+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":64,"up_from":120,"up_thru":57,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6819","nonce":3954360940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6821","nonce":3954360940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6825","nonce":3954360940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6823","nonce":3954360940}]},"public_addr":"172.21.15.181:6819/3954360940","cluster_addr":"172.21.15.181:6821/3954360940","heartbeat_back_addr":"172.21.15.181:6825/3954360940","heartbeat_front_addr":"172.21.15.181:6823/3954360940","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:40.299266+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:38.951721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.750591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:41.432577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:25.617972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.528356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:36.369593+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:05.532437+0000","dead_epoch":65},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:32.203935+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:31.146020+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:38.091678+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:35.472667+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:43.081826+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:39.544307+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:31.549706+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:26.664633+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[6,11]},{"pgid":"2.7","osds":[13,15]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":8,"length":3},{"begin":13,"length":4},{"begin":20,"length":2},{"begin":23,"length":2},{"begin":26,"length":8},{"begin":35,"length":2},{"begin":39,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:03:43.757 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T16:03:43.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:03:44.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:44.068 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":154,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:03:43.406922+0000","last_up_change":"2024-09-17T16:03:06.175857+0000","last_in_change":"2024-09-17T16:03:11.183940+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":64,"up_from":120,"up_thru":57,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6819","nonce":3954360940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6821","nonce":3954360940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6825","nonce":3954360940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6823","nonce":3954360940}]},"public_addr":"172.21.15.181:6819/3954360940","cluster_addr":"172.21.15.181:6821/3954360940","heartbeat_back_addr":"172.21.15.181:6825/3954360940","heartbeat_front_addr":"172.21.15.181:6823/3954360940","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:40.299266+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:38.951721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.750591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:41.432577+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:25.617972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.528356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:36.369593+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:05.532437+0000","dead_epoch":65},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:32.203935+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:31.146020+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:38.091678+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:35.472667+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:43.081826+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:39.544307+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:31.549706+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:26.664633+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[6,11]},{"pgid":"2.7","osds":[13,15]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":8,"length":3},{"begin":13,"length":4},{"begin":20,"length":2},{"begin":23,"length":2},{"begin":26,"length":8},{"begin":35,"length":2},{"begin":39,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:03:44.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T16:03:44.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T16:03:44.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T16:03:44.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T16:03:44.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T16:03:44.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T16:03:44.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-17T16:03:44.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-17T16:03:44.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-17T16:03:44.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-17T16:03:44.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-17T16:03:44.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-17T16:03:44.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-17T16:03:44.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-17T16:03:44.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-17T16:03:44.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-17T16:03:44.482 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-17T16:03:44.521 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-17T16:03:44.536 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T16:03:44.539 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T16:03:44.561 INFO:teuthology.orchestra.run.smithi006.stdout:515396075529 2024-09-17T16:03:44.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T16:03:44.571 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T16:03:44.583 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-17T16:03:44.592 INFO:teuthology.orchestra.run.smithi006.stdout:90194313251 2024-09-17T16:03:44.593 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T16:03:44.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T16:03:44.595 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T16:03:44.619 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T16:03:44.637 INFO:teuthology.orchestra.run.smithi006.stdout:94489280544 2024-09-17T16:03:44.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-17T16:03:44.638 INFO:teuthology.orchestra.run.smithi006.stdout:98784247842 2024-09-17T16:03:44.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-17T16:03:44.640 INFO:teuthology.orchestra.run.smithi006.stdout:94489280544 2024-09-17T16:03:44.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-17T16:03:44.680 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:44.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-17T16:03:44.715 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:44.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-17T16:03:45.592 INFO:teuthology.orchestra.run.smithi006.stdout:98784247842 2024-09-17T16:03:45.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247842 got 98784247842 for osd.12 2024-09-17T16:03:45.624 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.641 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.15 2024-09-17T16:03:45.662 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.676 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.677 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.2 2024-09-17T16:03:45.722 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.5 2024-09-17T16:03:45.728 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.767 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.769 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:45.802 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280545 for osd.11 2024-09-17T16:03:45.802 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.808 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.13 2024-09-17T16:03:45.811 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.815 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.816 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.823 INFO:teuthology.orchestra.run.smithi006.stdout:94489280547 2024-09-17T16:03:45.825 INFO:teuthology.orchestra.run.smithi006.stdout:515396075530 2024-09-17T16:03:45.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.4 2024-09-17T16:03:45.833 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.8 2024-09-17T16:03:45.842 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.849 INFO:teuthology.orchestra.run.smithi006.stdout:90194313251 2024-09-17T16:03:45.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280546 got 94489280547 for osd.14 2024-09-17T16:03:45.850 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 515396075529 got 515396075530 for osd.7 2024-09-17T16:03:45.853 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.855 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.10 2024-09-17T16:03:45.855 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.866 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313251 got 90194313251 for osd.1 2024-09-17T16:03:45.866 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.869 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.0 2024-09-17T16:03:45.882 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.899 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.917 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.6 2024-09-17T16:03:45.917 INFO:teuthology.orchestra.run.smithi006.stdout:94489280546 2024-09-17T16:03:45.917 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.921 INFO:teuthology.orchestra.run.smithi006.stdout:94489280545 2024-09-17T16:03:45.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280546 for osd.3 2024-09-17T16:03:45.931 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280545 for osd.9 2024-09-17T16:03:45.935 DEBUG:teuthology.parallel:result is None 2024-09-17T16:03:45.936 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-17T16:03:45.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:46.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:46.280 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:46.291 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-17T16:03:44.970283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529128,"kb_used_data":28588,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509420312,"statfs":{"total":1546188226560,"available":1545646399488,"internally_reserved":0,"allocated":29274112,"data_stored":23528808,"data_compressed":6537487,"data_compressed_allocated":8720384,"data_compressed_original":18317312,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.423913+0000","last_change":"2024-09-17T16:03:34.837672+0000","last_active":"2024-09-17T16:03:44.423913+0000","last_peered":"2024-09-17T16:03:44.423913+0000","last_clean":"2024-09-17T16:03:44.423913+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:44.423913+0000","last_undegraded":"2024-09-17T16:03:44.423913+0000","last_fullsized":"2024-09-17T16:03:44.423913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:34.837594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:34.837594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.149172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.531416+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:03:43.531416+0000","last_peered":"2024-09-17T16:03:43.531416+0000","last_clean":"2024-09-17T16:03:43.531416+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:43.531416+0000","last_undegraded":"2024-09-17T16:03:43.531416+0000","last_fullsized":"2024-09-17T16:03:43.531416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:03:24.594304+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:24.594208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:24.594208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:45.128978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.410670+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:03:43.410670+0000","last_peered":"2024-09-17T16:03:43.410670+0000","last_clean":"2024-09-17T16:03:43.410670+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:43.410670+0000","last_undegraded":"2024-09-17T16:03:43.410670+0000","last_fullsized":"2024-09-17T16:03:43.410670+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661247+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:03:44.661247+0000","last_peered":"2024-09-17T16:03:44.661247+0000","last_clean":"2024-09-17T16:03:44.661247+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:44.661247+0000","last_undegraded":"2024-09-17T16:03:44.661247+0000","last_fullsized":"2024-09-17T16:03:44.661247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661120+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:03:44.661120+0000","last_peered":"2024-09-17T16:03:44.661120+0000","last_clean":"2024-09-17T16:03:44.661120+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:44.661120+0000","last_undegraded":"2024-09-17T16:03:44.661120+0000","last_fullsized":"2024-09-17T16:03:44.661120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657246+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:03:44.657246+0000","last_peered":"2024-09-17T16:03:44.657246+0000","last_clean":"2024-09-17T16:03:44.657246+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:44.657246+0000","last_undegraded":"2024-09-17T16:03:44.657246+0000","last_fullsized":"2024-09-17T16:03:44.657246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":409,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657240+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:03:44.657240+0000","last_peered":"2024-09-17T16:03:44.657240+0000","last_clean":"2024-09-17T16:03:44.657240+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:44.657240+0000","last_undegraded":"2024-09-17T16:03:44.657240+0000","last_fullsized":"2024-09-17T16:03:44.657240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247842,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32956,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1810432,"data_stored":1395224,"data_compressed":404379,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28808,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30280,"kb_used_data":1780,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341560,"statfs":{"total":96636764160,"available":96605757440,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:46.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:46.594 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:46.595 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:46.607 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-17T16:03:44.970283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529128,"kb_used_data":28588,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509420312,"statfs":{"total":1546188226560,"available":1545646399488,"internally_reserved":0,"allocated":29274112,"data_stored":23528808,"data_compressed":6537487,"data_compressed_allocated":8720384,"data_compressed_original":18317312,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.423913+0000","last_change":"2024-09-17T16:03:34.837672+0000","last_active":"2024-09-17T16:03:44.423913+0000","last_peered":"2024-09-17T16:03:44.423913+0000","last_clean":"2024-09-17T16:03:44.423913+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:44.423913+0000","last_undegraded":"2024-09-17T16:03:44.423913+0000","last_fullsized":"2024-09-17T16:03:44.423913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:34.837594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:34.837594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.149172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.531416+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:03:43.531416+0000","last_peered":"2024-09-17T16:03:43.531416+0000","last_clean":"2024-09-17T16:03:43.531416+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:43.531416+0000","last_undegraded":"2024-09-17T16:03:43.531416+0000","last_fullsized":"2024-09-17T16:03:43.531416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:03:24.594304+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:24.594208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:24.594208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:45.128978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.410670+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:03:43.410670+0000","last_peered":"2024-09-17T16:03:43.410670+0000","last_clean":"2024-09-17T16:03:43.410670+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:43.410670+0000","last_undegraded":"2024-09-17T16:03:43.410670+0000","last_fullsized":"2024-09-17T16:03:43.410670+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661247+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:03:44.661247+0000","last_peered":"2024-09-17T16:03:44.661247+0000","last_clean":"2024-09-17T16:03:44.661247+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:44.661247+0000","last_undegraded":"2024-09-17T16:03:44.661247+0000","last_fullsized":"2024-09-17T16:03:44.661247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661120+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:03:44.661120+0000","last_peered":"2024-09-17T16:03:44.661120+0000","last_clean":"2024-09-17T16:03:44.661120+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:44.661120+0000","last_undegraded":"2024-09-17T16:03:44.661120+0000","last_fullsized":"2024-09-17T16:03:44.661120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657246+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:03:44.657246+0000","last_peered":"2024-09-17T16:03:44.657246+0000","last_clean":"2024-09-17T16:03:44.657246+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:44.657246+0000","last_undegraded":"2024-09-17T16:03:44.657246+0000","last_fullsized":"2024-09-17T16:03:44.657246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":409,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657240+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:03:44.657240+0000","last_peered":"2024-09-17T16:03:44.657240+0000","last_clean":"2024-09-17T16:03:44.657240+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:44.657240+0000","last_undegraded":"2024-09-17T16:03:44.657240+0000","last_fullsized":"2024-09-17T16:03:44.657240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247842,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32956,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1810432,"data_stored":1395224,"data_compressed":404379,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28808,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30280,"kb_used_data":1780,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341560,"statfs":{"total":96636764160,"available":96605757440,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:46.608 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-17T16:03:46.608 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T16:03:46.617 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T16:03:46.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:46.919 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:46.931 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-17T16:03:44.970283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529128,"kb_used_data":28588,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509420312,"statfs":{"total":1546188226560,"available":1545646399488,"internally_reserved":0,"allocated":29274112,"data_stored":23528808,"data_compressed":6537487,"data_compressed_allocated":8720384,"data_compressed_original":18317312,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.423913+0000","last_change":"2024-09-17T16:03:34.837672+0000","last_active":"2024-09-17T16:03:44.423913+0000","last_peered":"2024-09-17T16:03:44.423913+0000","last_clean":"2024-09-17T16:03:44.423913+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:44.423913+0000","last_undegraded":"2024-09-17T16:03:44.423913+0000","last_fullsized":"2024-09-17T16:03:44.423913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:34.837594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:34.837594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.149172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.531416+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:03:43.531416+0000","last_peered":"2024-09-17T16:03:43.531416+0000","last_clean":"2024-09-17T16:03:43.531416+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:43.531416+0000","last_undegraded":"2024-09-17T16:03:43.531416+0000","last_fullsized":"2024-09-17T16:03:43.531416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:03:24.594304+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:24.594208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:24.594208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:45.128978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.410670+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:03:43.410670+0000","last_peered":"2024-09-17T16:03:43.410670+0000","last_clean":"2024-09-17T16:03:43.410670+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:43.410670+0000","last_undegraded":"2024-09-17T16:03:43.410670+0000","last_fullsized":"2024-09-17T16:03:43.410670+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661247+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:03:44.661247+0000","last_peered":"2024-09-17T16:03:44.661247+0000","last_clean":"2024-09-17T16:03:44.661247+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:44.661247+0000","last_undegraded":"2024-09-17T16:03:44.661247+0000","last_fullsized":"2024-09-17T16:03:44.661247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661120+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:03:44.661120+0000","last_peered":"2024-09-17T16:03:44.661120+0000","last_clean":"2024-09-17T16:03:44.661120+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:44.661120+0000","last_undegraded":"2024-09-17T16:03:44.661120+0000","last_fullsized":"2024-09-17T16:03:44.661120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657246+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:03:44.657246+0000","last_peered":"2024-09-17T16:03:44.657246+0000","last_clean":"2024-09-17T16:03:44.657246+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:44.657246+0000","last_undegraded":"2024-09-17T16:03:44.657246+0000","last_fullsized":"2024-09-17T16:03:44.657246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":409,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657240+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:03:44.657240+0000","last_peered":"2024-09-17T16:03:44.657240+0000","last_clean":"2024-09-17T16:03:44.657240+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:44.657240+0000","last_undegraded":"2024-09-17T16:03:44.657240+0000","last_fullsized":"2024-09-17T16:03:44.657240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247842,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32956,"kb_used_data":1768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338884,"statfs":{"total":96636764160,"available":96603017216,"internally_reserved":0,"allocated":1810432,"data_stored":1395224,"data_compressed":404379,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28808,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30280,"kb_used_data":1780,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341560,"statfs":{"total":96636764160,"available":96605757440,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:46.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:47.236 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:47.236 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:47.248 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-17T16:03:46.970745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529160,"kb_used_data":28612,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509420280,"statfs":{"total":1546188226560,"available":1545646366720,"internally_reserved":0,"allocated":29298688,"data_stored":23548444,"data_compressed":6543676,"data_compressed_allocated":8728576,"data_compressed_original":18333696,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.569229"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.423913+0000","last_change":"2024-09-17T16:03:34.837672+0000","last_active":"2024-09-17T16:03:44.423913+0000","last_peered":"2024-09-17T16:03:44.423913+0000","last_clean":"2024-09-17T16:03:44.423913+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:44.423913+0000","last_undegraded":"2024-09-17T16:03:44.423913+0000","last_fullsized":"2024-09-17T16:03:44.423913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:34.837594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:34.837594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.149172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.531416+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:03:43.531416+0000","last_peered":"2024-09-17T16:03:43.531416+0000","last_clean":"2024-09-17T16:03:43.531416+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:43.531416+0000","last_undegraded":"2024-09-17T16:03:43.531416+0000","last_fullsized":"2024-09-17T16:03:43.531416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:03:24.594304+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:24.594208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:24.594208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:45.128978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.410670+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:03:43.410670+0000","last_peered":"2024-09-17T16:03:43.410670+0000","last_clean":"2024-09-17T16:03:43.410670+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:43.410670+0000","last_undegraded":"2024-09-17T16:03:43.410670+0000","last_fullsized":"2024-09-17T16:03:43.410670+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661247+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:03:44.661247+0000","last_peered":"2024-09-17T16:03:44.661247+0000","last_clean":"2024-09-17T16:03:44.661247+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:44.661247+0000","last_undegraded":"2024-09-17T16:03:44.661247+0000","last_fullsized":"2024-09-17T16:03:44.661247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661120+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:03:44.661120+0000","last_peered":"2024-09-17T16:03:44.661120+0000","last_clean":"2024-09-17T16:03:44.661120+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:44.661120+0000","last_undegraded":"2024-09-17T16:03:44.661120+0000","last_fullsized":"2024-09-17T16:03:44.661120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657246+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:03:44.657246+0000","last_peered":"2024-09-17T16:03:44.657246+0000","last_clean":"2024-09-17T16:03:44.657246+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:44.657246+0000","last_undegraded":"2024-09-17T16:03:44.657246+0000","last_fullsized":"2024-09-17T16:03:44.657246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":409,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657240+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:03:44.657240+0000","last_peered":"2024-09-17T16:03:44.657240+0000","last_clean":"2024-09-17T16:03:44.657240+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:44.657240+0000","last_undegraded":"2024-09-17T16:03:44.657240+0000","last_fullsized":"2024-09-17T16:03:44.657240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32988,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1835008,"data_stored":1414860,"data_compressed":410568,"data_compressed_allocated":544768,"data_compressed_original":1089536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28808,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30280,"kb_used_data":1780,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341560,"statfs":{"total":96636764160,"available":96605757440,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:47.249 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T16:03:47.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:47.551 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:47.552 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:47.564 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-17T16:03:47.444760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529160,"kb_used_data":28612,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509420280,"statfs":{"total":1546188226560,"available":1545646366720,"internally_reserved":0,"allocated":29298688,"data_stored":23548444,"data_compressed":6543676,"data_compressed_allocated":8728576,"data_compressed_original":18333696,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.041383"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":205,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.423913+0000","last_change":"2024-09-17T16:03:34.837672+0000","last_active":"2024-09-17T16:03:44.423913+0000","last_peered":"2024-09-17T16:03:44.423913+0000","last_clean":"2024-09-17T16:03:44.423913+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:44.423913+0000","last_undegraded":"2024-09-17T16:03:44.423913+0000","last_fullsized":"2024-09-17T16:03:44.423913+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:34.837594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:34.837594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.149172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.531416+0000","last_change":"2024-09-17T16:02:48.683611+0000","last_active":"2024-09-17T16:03:43.531416+0000","last_peered":"2024-09-17T16:03:43.531416+0000","last_clean":"2024-09-17T16:03:43.531416+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:43.531416+0000","last_undegraded":"2024-09-17T16:03:43.531416+0000","last_fullsized":"2024-09-17T16:03:43.531416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:48.683519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:48.683519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:02.813516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":274,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:03:24.594304+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:24.594208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:24.594208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:45.128978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":269,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.410670+0000","last_change":"2024-09-17T16:02:37.368605+0000","last_active":"2024-09-17T16:03:43.410670+0000","last_peered":"2024-09-17T16:03:43.410670+0000","last_clean":"2024-09-17T16:03:43.410670+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:43.410670+0000","last_undegraded":"2024-09-17T16:03:43.410670+0000","last_fullsized":"2024-09-17T16:03:43.410670+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:37.368543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:37.368543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":279,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661247+0000","last_change":"2024-09-17T16:02:35.421658+0000","last_active":"2024-09-17T16:03:44.661247+0000","last_peered":"2024-09-17T16:03:44.661247+0000","last_clean":"2024-09-17T16:03:44.661247+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:44.661247+0000","last_undegraded":"2024-09-17T16:03:44.661247+0000","last_fullsized":"2024-09-17T16:03:44.661247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-17T16:02:35.421589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:35.421589+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:01.339912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-17T16:03:43.768995+0000","last_change":"2024-09-17T16:02:30.690901+0000","last_active":"2024-09-17T16:03:43.768995+0000","last_peered":"2024-09-17T16:03:43.768995+0000","last_clean":"2024-09-17T16:03:43.768995+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:43.768995+0000","last_undegraded":"2024-09-17T16:03:43.768995+0000","last_fullsized":"2024-09-17T16:03:43.768995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":277,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.661120+0000","last_change":"2024-09-17T16:02:40.404078+0000","last_active":"2024-09-17T16:03:44.661120+0000","last_peered":"2024-09-17T16:03:44.661120+0000","last_clean":"2024-09-17T16:03:44.661120+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:44.661120+0000","last_undegraded":"2024-09-17T16:03:44.661120+0000","last_fullsized":"2024-09-17T16:03:44.661120+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:02:40.404002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.404002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:07.681949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":267,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657246+0000","last_change":"2024-09-17T16:01:21.331436+0000","last_active":"2024-09-17T16:03:44.657246+0000","last_peered":"2024-09-17T16:03:44.657246+0000","last_clean":"2024-09-17T16:03:44.657246+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:44.657246+0000","last_undegraded":"2024-09-17T16:03:44.657246+0000","last_fullsized":"2024-09-17T16:03:44.657246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:01:17.738476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":409,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-17T16:03:44.657240+0000","last_change":"2024-09-17T16:02:40.347624+0000","last_active":"2024-09-17T16:03:44.657240+0000","last_peered":"2024-09-17T16:03:44.657240+0000","last_clean":"2024-09-17T16:03:44.657240+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:44.657240+0000","last_undegraded":"2024-09-17T16:03:44.657240+0000","last_fullsized":"2024-09-17T16:03:44.657240+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:02:40.347549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:02:40.347549+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32988,"kb_used_data":1792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1835008,"data_stored":1414860,"data_compressed":410568,"data_compressed_allocated":544768,"data_compressed_original":1089536,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33024,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338816,"statfs":{"total":96636764160,"available":96602947584,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28808,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343032,"statfs":{"total":96636764160,"available":96607264768,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280546,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32968,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338872,"statfs":{"total":96636764160,"available":96603004928,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30280,"kb_used_data":1780,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341560,"statfs":{"total":96636764160,"available":96605757440,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1405065,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1822720,"data_stored":1405046,"data_compressed":407474,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1880064,"data_stored":1933974,"data_compressed":417973,"data_compressed_allocated":577536,"data_compressed_original":1593344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:47.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T16:03:47.872 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:47.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":156,"fsid":"4345ca04-2628-4bef-a537-6837268b96c9","created":"2024-09-17T16:01:02.569356+0000","modified":"2024-09-17T16:03:47.437016+0000","last_up_change":"2024-09-17T16:03:06.175857+0000","last_in_change":"2024-09-17T16:03:11.183940+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T16:01:14.963100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T16:01:17.236630+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"282c4269-7e53-43fc-af4b-3b133d82837c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6809","nonce":865094279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6811","nonce":865094279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6815","nonce":865094279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":865094279},{"type":"v1","addr":"172.21.15.6:6813","nonce":865094279}]},"public_addr":"172.21.15.6:6809/865094279","cluster_addr":"172.21.15.6:6811/865094279","heartbeat_back_addr":"172.21.15.6:6815/865094279","heartbeat_front_addr":"172.21.15.6:6813/865094279","state":["exists","up"]},{"osd":1,"uuid":"66e20d10-2bb0-4130-93a5-45bcdd2de605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6803","nonce":1460756480}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6807","nonce":1460756480}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6813","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6815","nonce":1460756480}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":1460756480},{"type":"v1","addr":"172.21.15.50:6811","nonce":1460756480}]},"public_addr":"172.21.15.50:6803/1460756480","cluster_addr":"172.21.15.50:6807/1460756480","heartbeat_back_addr":"172.21.15.50:6815/1460756480","heartbeat_front_addr":"172.21.15.50:6811/1460756480","state":["exists","up"]},{"osd":2,"uuid":"c6ec9e20-133a-471b-b63c-6453a7bde86c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6817","nonce":1905413450}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6819","nonce":1905413450}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6823","nonce":1905413450}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1905413450},{"type":"v1","addr":"172.21.15.169:6821","nonce":1905413450}]},"public_addr":"172.21.15.169:6817/1905413450","cluster_addr":"172.21.15.169:6819/1905413450","heartbeat_back_addr":"172.21.15.169:6823/1905413450","heartbeat_front_addr":"172.21.15.169:6821/1905413450","state":["exists","up"]},{"osd":3,"uuid":"3491a731-ca96-4981-be73-e005e00c315f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6811","nonce":2918557179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6813","nonce":2918557179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6817","nonce":2918557179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2918557179},{"type":"v1","addr":"172.21.15.181:6815","nonce":2918557179}]},"public_addr":"172.21.15.181:6811/2918557179","cluster_addr":"172.21.15.181:6813/2918557179","heartbeat_back_addr":"172.21.15.181:6817/2918557179","heartbeat_front_addr":"172.21.15.181:6815/2918557179","state":["exists","up"]},{"osd":4,"uuid":"3bdfaa34-bc10-4cec-b5c2-86e05d444ffd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6801","nonce":2257685015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6803","nonce":2257685015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6807","nonce":2257685015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":2257685015},{"type":"v1","addr":"172.21.15.6:6805","nonce":2257685015}]},"public_addr":"172.21.15.6:6801/2257685015","cluster_addr":"172.21.15.6:6803/2257685015","heartbeat_back_addr":"172.21.15.6:6807/2257685015","heartbeat_front_addr":"172.21.15.6:6805/2257685015","state":["exists","up"]},{"osd":5,"uuid":"5f7bc481-65a7-45bb-b3e9-a8d3840b5bb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6825","nonce":2399951648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6827","nonce":2399951648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6831","nonce":2399951648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2399951648},{"type":"v1","addr":"172.21.15.50:6829","nonce":2399951648}]},"public_addr":"172.21.15.50:6825/2399951648","cluster_addr":"172.21.15.50:6827/2399951648","heartbeat_back_addr":"172.21.15.50:6831/2399951648","heartbeat_front_addr":"172.21.15.50:6829/2399951648","state":["exists","up"]},{"osd":6,"uuid":"ddd3edea-260a-4ec9-8533-841ed933fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6825","nonce":1225429423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6827","nonce":1225429423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6831","nonce":1225429423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1225429423},{"type":"v1","addr":"172.21.15.169:6829","nonce":1225429423}]},"public_addr":"172.21.15.169:6825/1225429423","cluster_addr":"172.21.15.169:6827/1225429423","heartbeat_back_addr":"172.21.15.169:6831/1225429423","heartbeat_front_addr":"172.21.15.169:6829/1225429423","state":["exists","up"]},{"osd":7,"uuid":"7afc1a18-98c2-4a58-a09d-1b4f474d4dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":64,"up_from":120,"up_thru":57,"down_at":65,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6819","nonce":3954360940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6821","nonce":3954360940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6825","nonce":3954360940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3954360940},{"type":"v1","addr":"172.21.15.181:6823","nonce":3954360940}]},"public_addr":"172.21.15.181:6819/3954360940","cluster_addr":"172.21.15.181:6821/3954360940","heartbeat_back_addr":"172.21.15.181:6825/3954360940","heartbeat_front_addr":"172.21.15.181:6823/3954360940","state":["exists","up"]},{"osd":8,"uuid":"3e45f80f-856b-4d3f-b002-255f16a14f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6825","nonce":2216439459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6827","nonce":2216439459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6831","nonce":2216439459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":2216439459},{"type":"v1","addr":"172.21.15.6:6829","nonce":2216439459}]},"public_addr":"172.21.15.6:6825/2216439459","cluster_addr":"172.21.15.6:6827/2216439459","heartbeat_back_addr":"172.21.15.6:6831/2216439459","heartbeat_front_addr":"172.21.15.6:6829/2216439459","state":["exists","up"]},{"osd":9,"uuid":"b415c423-24de-4d86-bec4-ea330364ff96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6817","nonce":1727176716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6819","nonce":1727176716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6823","nonce":1727176716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":1727176716},{"type":"v1","addr":"172.21.15.50:6821","nonce":1727176716}]},"public_addr":"172.21.15.50:6817/1727176716","cluster_addr":"172.21.15.50:6819/1727176716","heartbeat_back_addr":"172.21.15.50:6823/1727176716","heartbeat_front_addr":"172.21.15.50:6821/1727176716","state":["exists","up"]},{"osd":10,"uuid":"e471e1cb-9c0b-4c99-8622-59eb7956770d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6809","nonce":3604183526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6811","nonce":3604183526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6815","nonce":3604183526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3604183526},{"type":"v1","addr":"172.21.15.169:6813","nonce":3604183526}]},"public_addr":"172.21.15.169:6809/3604183526","cluster_addr":"172.21.15.169:6811/3604183526","heartbeat_back_addr":"172.21.15.169:6815/3604183526","heartbeat_front_addr":"172.21.15.169:6813/3604183526","state":["exists","up"]},{"osd":11,"uuid":"c2bf4255-7ce1-4673-bedf-fd055d841fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6827","nonce":3420013159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6829","nonce":3420013159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6833","nonce":3420013159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3420013159},{"type":"v1","addr":"172.21.15.181:6831","nonce":3420013159}]},"public_addr":"172.21.15.181:6827/3420013159","cluster_addr":"172.21.15.181:6829/3420013159","heartbeat_back_addr":"172.21.15.181:6833/3420013159","heartbeat_front_addr":"172.21.15.181:6831/3420013159","state":["exists","up"]},{"osd":12,"uuid":"4c16f3ee-f427-41d5-b0ba-9e676ddbe032","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6817","nonce":2310487797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6819","nonce":2310487797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6823","nonce":2310487797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":2310487797},{"type":"v1","addr":"172.21.15.6:6821","nonce":2310487797}]},"public_addr":"172.21.15.6:6817/2310487797","cluster_addr":"172.21.15.6:6819/2310487797","heartbeat_back_addr":"172.21.15.6:6823/2310487797","heartbeat_front_addr":"172.21.15.6:6821/2310487797","state":["exists","up"]},{"osd":13,"uuid":"0a4386aa-1a59-44c8-8d1e-7d43bf10de2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6802","nonce":1210437211}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6806","nonce":1210437211}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6814","nonce":1210437211}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":1210437211},{"type":"v1","addr":"172.21.15.50:6810","nonce":1210437211}]},"public_addr":"172.21.15.50:6802/1210437211","cluster_addr":"172.21.15.50:6806/1210437211","heartbeat_back_addr":"172.21.15.50:6814/1210437211","heartbeat_front_addr":"172.21.15.50:6810/1210437211","state":["exists","up"]},{"osd":14,"uuid":"4c54e35c-e226-4d8c-bad0-51dd1cd95fe0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6801","nonce":4170280864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6803","nonce":4170280864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6807","nonce":4170280864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":4170280864},{"type":"v1","addr":"172.21.15.169:6805","nonce":4170280864}]},"public_addr":"172.21.15.169:6801/4170280864","cluster_addr":"172.21.15.169:6803/4170280864","heartbeat_back_addr":"172.21.15.169:6807/4170280864","heartbeat_front_addr":"172.21.15.169:6805/4170280864","state":["exists","up"]},{"osd":15,"uuid":"0281e750-f74a-4760-bbef-fbfa4127d06b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6803","nonce":2815528414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6805","nonce":2815528414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6809","nonce":2815528414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2815528414},{"type":"v1","addr":"172.21.15.181:6807","nonce":2815528414}]},"public_addr":"172.21.15.181:6803/2815528414","cluster_addr":"172.21.15.181:6805/2815528414","heartbeat_back_addr":"172.21.15.181:6809/2815528414","heartbeat_front_addr":"172.21.15.181:6807/2815528414","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:40.299266+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:38.951721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.750591+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:46.683054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:44.086196+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:37.528356+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:36.369593+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:05.532437+0000","dead_epoch":65},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:32.203935+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:31.146020+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:38.091678+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:35.472667+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:03:43.081826+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:39.544307+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:31.549706+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T16:02:26.664633+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.1","osds":[6,11]},{"pgid":"2.7","osds":[13,15]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":2,"length":1},{"begin":8,"length":3},{"begin":13,"length":4},{"begin":20,"length":2},{"begin":23,"length":2},{"begin":26,"length":8},{"begin":35,"length":2},{"begin":39,"length":1},{"begin":41,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T16:03:48.888 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T16:03:48.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:49.039 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:49.039 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:49.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T16:03:49.353 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T16:03:49.365 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T16:03:49.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:49.504 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:49.505 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:49.505 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:49.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T16:03:49.821 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T16:03:49.834 INFO:tasks.ceph:Scrubbing osd.8 2024-09-17T16:03:49.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:49.970 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:49.971 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:49.971 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:49.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-17T16:03:50.284 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 8 to deep-scrub 2024-09-17T16:03:50.297 INFO:tasks.ceph:Scrubbing osd.12 2024-09-17T16:03:50.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:50.434 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:50.434 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:50.434 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:50.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-17T16:03:50.748 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 12 to deep-scrub 2024-09-17T16:03:50.761 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T16:03:50.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:50.898 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:50.899 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:50.899 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:50.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T16:03:51.212 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T16:03:51.225 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T16:03:51.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:51.362 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:51.362 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:51.362 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:51.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T16:03:51.681 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T16:03:51.693 INFO:tasks.ceph:Scrubbing osd.9 2024-09-17T16:03:51.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:51.831 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:51.831 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:51.831 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:51.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-17T16:03:52.147 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 9 to deep-scrub 2024-09-17T16:03:52.160 INFO:tasks.ceph:Scrubbing osd.13 2024-09-17T16:03:52.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:52.301 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:52.301 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:52.302 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:52.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-17T16:03:52.615 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 13 to deep-scrub 2024-09-17T16:03:52.628 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T16:03:52.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:52.767 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:52.768 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:52.768 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:52.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T16:03:53.086 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T16:03:53.099 INFO:tasks.ceph:Scrubbing osd.6 2024-09-17T16:03:53.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:53.236 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:53.236 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:53.236 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:53.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-17T16:03:53.553 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 6 to deep-scrub 2024-09-17T16:03:53.566 INFO:tasks.ceph:Scrubbing osd.10 2024-09-17T16:03:53.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:53.704 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:53.704 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:53.704 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:53.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-17T16:03:54.021 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 10 to deep-scrub 2024-09-17T16:03:54.033 INFO:tasks.ceph:Scrubbing osd.14 2024-09-17T16:03:54.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:54.172 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:54.172 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:54.172 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:54.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-17T16:03:54.488 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 14 to deep-scrub 2024-09-17T16:03:54.502 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T16:03:54.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:54.640 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:54.640 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:54.640 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:54.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T16:03:54.957 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T16:03:54.970 INFO:tasks.ceph:Scrubbing osd.7 2024-09-17T16:03:54.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:55.107 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:55.108 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:55.108 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:55.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-17T16:03:55.420 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 7 to deep-scrub 2024-09-17T16:03:55.433 INFO:tasks.ceph:Scrubbing osd.11 2024-09-17T16:03:55.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:55.571 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:55.571 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:55.571 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:55.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-17T16:03:55.886 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 11 to deep-scrub 2024-09-17T16:03:55.899 INFO:tasks.ceph:Scrubbing osd.15 2024-09-17T16:03:55.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-17T16:03:56.041 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-09-17T16:03:56.042 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T16:03:56.042 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-09-17T16:03:56.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-17T16:03:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 15 to deep-scrub 2024-09-17T16:03:56.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:03:56.673 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:03:56.674 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:03:56.686 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-17T16:03:56.472467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":529664,"kb_used_data":29176,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509419776,"statfs":{"total":1546188226560,"available":1545645850624,"internally_reserved":0,"allocated":29876224,"data_stored":24009702,"data_compressed":6689094,"data_compressed_allocated":8921088,"data_compressed_original":18718720,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.018655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:52.933274+0000","last_change":"2024-09-17T16:03:52.933274+0000","last_active":"2024-09-17T16:03:52.933274+0000","last_peered":"2024-09-17T16:03:52.933274+0000","last_clean":"2024-09-17T16:03:52.933274+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:52.933274+0000","last_undegraded":"2024-09-17T16:03:52.933274+0000","last_fullsized":"2024-09-17T16:03:52.933274+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:52.933196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:52.933196+0000","last_clean_scrub_stamp":"2024-09-17T16:03:52.933196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:18.859431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":287,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:52.589016+0000","last_change":"2024-09-17T16:03:52.589016+0000","last_active":"2024-09-17T16:03:52.589016+0000","last_peered":"2024-09-17T16:03:52.589016+0000","last_clean":"2024-09-17T16:03:52.589016+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:52.589016+0000","last_undegraded":"2024-09-17T16:03:52.589016+0000","last_fullsized":"2024-09-17T16:03:52.589016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:52.588911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:52.588911+0000","last_clean_scrub_stamp":"2024-09-17T16:03:52.588911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.204186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":288,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:54.740375+0000","last_change":"2024-09-17T16:03:54.740375+0000","last_active":"2024-09-17T16:03:54.740375+0000","last_peered":"2024-09-17T16:03:54.740375+0000","last_clean":"2024-09-17T16:03:54.740375+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:54.740375+0000","last_undegraded":"2024-09-17T16:03:54.740375+0000","last_fullsized":"2024-09-17T16:03:54.740375+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:54.740276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:54.740276+0000","last_clean_scrub_stamp":"2024-09-17T16:03:54.740276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:08.782144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":291,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:51.515634+0000","last_change":"2024-09-17T16:03:51.515634+0000","last_active":"2024-09-17T16:03:51.515634+0000","last_peered":"2024-09-17T16:03:51.515634+0000","last_clean":"2024-09-17T16:03:51.515634+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:51.515634+0000","last_undegraded":"2024-09-17T16:03:51.515634+0000","last_fullsized":"2024-09-17T16:03:51.515634+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:51.515549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:51.515549+0000","last_clean_scrub_stamp":"2024-09-17T16:03:51.515549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:17.060405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'3","reported_seq":292,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:51.365803+0000","last_change":"2024-09-17T16:03:51.365803+0000","last_active":"2024-09-17T16:03:51.365803+0000","last_peered":"2024-09-17T16:03:51.365803+0000","last_clean":"2024-09-17T16:03:51.365803+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:51.365803+0000","last_undegraded":"2024-09-17T16:03:51.365803+0000","last_fullsized":"2024-09-17T16:03:51.365803+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'3","last_scrub_stamp":"2024-09-17T16:03:51.365717+0000","last_deep_scrub":"157'3","last_deep_scrub_stamp":"2024-09-17T16:03:51.365717+0000","last_clean_scrub_stamp":"2024-09-17T16:03:51.365717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:17.009802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:53.775270+0000","last_change":"2024-09-17T16:03:53.775270+0000","last_active":"2024-09-17T16:03:53.775270+0000","last_peered":"2024-09-17T16:03:53.775270+0000","last_clean":"2024-09-17T16:03:53.775270+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:53.775270+0000","last_undegraded":"2024-09-17T16:03:53.775270+0000","last_fullsized":"2024-09-17T16:03:53.775270+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:53.775171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:53.775171+0000","last_clean_scrub_stamp":"2024-09-17T16:03:53.775171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:09.393204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:50.367095+0000","last_change":"2024-09-17T16:03:50.367095+0000","last_active":"2024-09-17T16:03:50.367095+0000","last_peered":"2024-09-17T16:03:50.367095+0000","last_clean":"2024-09-17T16:03:50.367095+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:50.367095+0000","last_undegraded":"2024-09-17T16:03:50.367095+0000","last_fullsized":"2024-09-17T16:03:50.367095+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:50.367028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:50.367028+0000","last_clean_scrub_stamp":"2024-09-17T16:03:50.367028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:07.148952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":279,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:48.757711+0000","last_change":"2024-09-17T16:03:45.111268+0000","last_active":"2024-09-17T16:03:48.757711+0000","last_peered":"2024-09-17T16:03:48.757711+0000","last_clean":"2024-09-17T16:03:48.757711+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:48.757711+0000","last_undegraded":"2024-09-17T16:03:48.757711+0000","last_fullsized":"2024-09-17T16:03:48.757711+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-17T16:03:45.111180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:17.738476+0000","last_clean_scrub_stamp":"2024-09-17T16:03:45.111180+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:56.836932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":421,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-17T16:03:49.166692+0000","last_change":"2024-09-17T16:03:49.166692+0000","last_active":"2024-09-17T16:03:49.166692+0000","last_peered":"2024-09-17T16:03:49.166692+0000","last_clean":"2024-09-17T16:03:49.166692+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:49.166692+0000","last_undegraded":"2024-09-17T16:03:49.166692+0000","last_fullsized":"2024-09-17T16:03:49.166692+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-17T16:03:49.166588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:01:15.727345+0000","last_clean_scrub_stamp":"2024-09-17T16:03:49.166588+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:07.165686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280549,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33008,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37160,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1916928,"data_stored":1963416,"data_compressed":427255,"data_compressed_allocated":589824,"data_compressed_original":1617920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28844,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342996,"statfs":{"total":96636764160,"available":96607227904,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33008,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280548,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33000,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338840,"statfs":{"total":96636764160,"available":96602972160,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075532,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30320,"kb_used_data":1816,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341520,"statfs":{"total":96636764160,"available":96605716480,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1863680,"data_stored":1434507,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33004,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1863680,"data_stored":1434507,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33004,"kb_used_data":1816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338836,"statfs":{"total":96636764160,"available":96602968064,"internally_reserved":0,"allocated":1859584,"data_stored":1434488,"data_compressed":416756,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33064,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338776,"statfs":{"total":96636764160,"available":96602906624,"internally_reserved":0,"allocated":1916928,"data_stored":1963416,"data_compressed":427255,"data_compressed_allocated":589824,"data_compressed_original":1617920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:03:56.690 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T16:03:45.111180+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=16, tm_min=3, tm_sec=45, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=16, tm_min=3, tm_sec=47, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T16:03:56.690 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T16:04:16.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T16:04:17.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-09-17T16:04:17.009 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-09-17T16:04:17.021 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-17T16:04:16.977613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":10,"kb":1509949440,"kb_used":530280,"kb_used_data":29752,"kb_used_omap":39,"kb_used_meta":500120,"kb_avail":1509419160,"statfs":{"total":1546188226560,"available":1545645219840,"internally_reserved":0,"allocated":30466048,"data_stored":24481574,"data_compressed":6837510,"data_compressed_allocated":9117696,"data_compressed_original":19111936,"omap_allocated":40409,"internal_metadata":512123431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002727"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.486168+0000","last_change":"2024-09-17T16:03:52.933274+0000","last_active":"2024-09-17T16:03:57.486168+0000","last_peered":"2024-09-17T16:03:57.486168+0000","last_clean":"2024-09-17T16:03:57.486168+0000","last_became_active":"2024-09-17T16:01:58.347411+0000","last_became_peered":"2024-09-17T16:01:58.347411+0000","last_unstale":"2024-09-17T16:03:57.486168+0000","last_undegraded":"2024-09-17T16:03:57.486168+0000","last_fullsized":"2024-09-17T16:03:57.486168+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:52.933196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:52.933196+0000","last_clean_scrub_stamp":"2024-09-17T16:03:52.933196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:18.859431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,15],"acting":[13,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":293,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:58.296094+0000","last_change":"2024-09-17T16:03:52.589016+0000","last_active":"2024-09-17T16:03:58.296094+0000","last_peered":"2024-09-17T16:03:58.296094+0000","last_clean":"2024-09-17T16:03:58.296094+0000","last_became_active":"2024-09-17T16:01:18.762593+0000","last_became_peered":"2024-09-17T16:01:18.762593+0000","last_unstale":"2024-09-17T16:03:58.296094+0000","last_undegraded":"2024-09-17T16:03:58.296094+0000","last_fullsized":"2024-09-17T16:03:58.296094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:52.588911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:52.588911+0000","last_clean_scrub_stamp":"2024-09-17T16:03:52.588911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:02.204186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045869999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6],"acting":[9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":293,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.490942+0000","last_change":"2024-09-17T16:03:54.740375+0000","last_active":"2024-09-17T16:03:57.490942+0000","last_peered":"2024-09-17T16:03:57.490942+0000","last_clean":"2024-09-17T16:03:57.490942+0000","last_became_active":"2024-09-17T16:01:18.785982+0000","last_became_peered":"2024-09-17T16:01:18.785982+0000","last_unstale":"2024-09-17T16:03:57.490942+0000","last_undegraded":"2024-09-17T16:03:57.490942+0000","last_fullsized":"2024-09-17T16:03:57.490942+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:54.740276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:54.740276+0000","last_clean_scrub_stamp":"2024-09-17T16:03:54.740276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:08.782144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041416800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.979958+0000","last_change":"2024-09-17T16:03:51.515634+0000","last_active":"2024-09-17T16:03:57.979958+0000","last_peered":"2024-09-17T16:03:57.979958+0000","last_clean":"2024-09-17T16:03:57.979958+0000","last_became_active":"2024-09-17T16:01:18.758059+0000","last_became_peered":"2024-09-17T16:01:18.758059+0000","last_unstale":"2024-09-17T16:03:57.979958+0000","last_undegraded":"2024-09-17T16:03:57.979958+0000","last_fullsized":"2024-09-17T16:03:57.979958+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:51.515549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:51.515549+0000","last_clean_scrub_stamp":"2024-09-17T16:03:51.515549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:17.060405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043610799999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"157'3","reported_seq":297,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.485673+0000","last_change":"2024-09-17T16:03:51.365803+0000","last_active":"2024-09-17T16:03:57.485673+0000","last_peered":"2024-09-17T16:03:57.485673+0000","last_clean":"2024-09-17T16:03:57.485673+0000","last_became_active":"2024-09-17T16:01:18.758052+0000","last_became_peered":"2024-09-17T16:01:18.758052+0000","last_unstale":"2024-09-17T16:03:57.485673+0000","last_undegraded":"2024-09-17T16:03:57.485673+0000","last_fullsized":"2024-09-17T16:03:57.485673+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"157'3","last_scrub_stamp":"2024-09-17T16:03:51.365717+0000","last_deep_scrub":"157'3","last_deep_scrub_stamp":"2024-09-17T16:03:51.365717+0000","last_clean_scrub_stamp":"2024-09-17T16:03:51.365717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:17.009802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096117499999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9],"acting":[8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.490939+0000","last_change":"2024-09-17T16:03:53.775270+0000","last_active":"2024-09-17T16:03:57.490939+0000","last_peered":"2024-09-17T16:03:57.490939+0000","last_clean":"2024-09-17T16:03:57.490939+0000","last_became_active":"2024-09-17T16:02:30.690530+0000","last_became_peered":"2024-09-17T16:02:30.690530+0000","last_unstale":"2024-09-17T16:03:57.490939+0000","last_undegraded":"2024-09-17T16:03:57.490939+0000","last_fullsized":"2024-09-17T16:03:57.490939+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:53.775171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:53.775171+0000","last_clean_scrub_stamp":"2024-09-17T16:03:53.775171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:09.393204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11],"acting":[6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":294,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:57.485664+0000","last_change":"2024-09-17T16:03:50.367095+0000","last_active":"2024-09-17T16:03:57.485664+0000","last_peered":"2024-09-17T16:03:57.485664+0000","last_clean":"2024-09-17T16:03:57.485664+0000","last_became_active":"2024-09-17T16:01:18.762861+0000","last_became_peered":"2024-09-17T16:01:18.762861+0000","last_unstale":"2024-09-17T16:03:57.485664+0000","last_undegraded":"2024-09-17T16:03:57.485664+0000","last_fullsized":"2024-09-17T16:03:57.485664+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T16:03:50.367028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T16:03:50.367028+0000","last_clean_scrub_stamp":"2024-09-17T16:03:50.367028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:07.148952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056053199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6],"acting":[8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.3","version":"158'2","reported_seq":294,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:58.296790+0000","last_change":"2024-09-17T16:03:56.070084+0000","last_active":"2024-09-17T16:03:58.296790+0000","last_peered":"2024-09-17T16:03:58.296790+0000","last_clean":"2024-09-17T16:03:58.296790+0000","last_became_active":"2024-09-17T16:01:18.998023+0000","last_became_peered":"2024-09-17T16:01:18.998023+0000","last_unstale":"2024-09-17T16:03:58.296790+0000","last_undegraded":"2024-09-17T16:03:58.296790+0000","last_fullsized":"2024-09-17T16:03:58.296790+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'2","last_scrub_stamp":"2024-09-17T16:03:56.069994+0000","last_deep_scrub":"158'2","last_deep_scrub_stamp":"2024-09-17T16:03:56.069994+0000","last_clean_scrub_stamp":"2024-09-17T16:03:56.069994+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:01.023152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00047514299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0],"acting":[11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"158'114","reported_seq":437,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-17T16:03:58.296784+0000","last_change":"2024-09-17T16:03:57.097835+0000","last_active":"2024-09-17T16:03:58.296784+0000","last_peered":"2024-09-17T16:03:58.296784+0000","last_clean":"2024-09-17T16:03:58.296784+0000","last_became_active":"2024-09-17T16:01:16.939727+0000","last_became_peered":"2024-09-17T16:01:16.939727+0000","last_unstale":"2024-09-17T16:03:58.296784+0000","last_undegraded":"2024-09-17T16:03:58.296784+0000","last_fullsized":"2024-09-17T16:03:58.296784+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"158'114","last_scrub_stamp":"2024-09-17T16:03:57.097755+0000","last_deep_scrub":"158'114","last_deep_scrub_stamp":"2024-09-17T16:03:57.097755+0000","last_clean_scrub_stamp":"2024-09-17T16:03:57.097755+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:05:23.787161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1057856,"data_compressed":20998,"data_compressed_allocated":73728,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":12,"up_from":23,"seq":98784247849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,8,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280553,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33040,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,8,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37200,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334640,"statfs":{"total":96636764160,"available":96598671360,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,4,7,8,9,10,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,5,6,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[4,5,6,7,8,10,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33104,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338736,"statfs":{"total":96636764160,"available":96602865664,"internally_reserved":0,"allocated":1953792,"data_stored":1992908,"data_compressed":436531,"data_compressed_allocated":602112,"data_compressed_original":1642496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28880,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342960,"statfs":{"total":96636764160,"available":96607191040,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33044,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":120,"seq":515396075536,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":30356,"kb_used_data":1852,"kb_used_omap":16,"kb_used_meta":28463,"kb_avail":94341484,"statfs":{"total":96636764160,"available":96605679616,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":16479,"internal_metadata":29147041},"hb_peers":[4,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1900544,"data_stored":1463999,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,5,6,7,9,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1900544,"data_stored":1463999,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,6,7,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33044,"kb_used_data":1852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338796,"statfs":{"total":96636764160,"available":96602927104,"internally_reserved":0,"allocated":1896448,"data_stored":1463980,"data_compressed":426032,"data_compressed_allocated":565248,"data_compressed_original":1130496,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[0,1,3,4,5,8,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1953792,"data_stored":1992908,"data_compressed":436531,"data_compressed_allocated":602112,"data_compressed_original":1642496,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,4,6,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":528928,"data_compressed":10499,"data_compressed_allocated":36864,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T16:04:17.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T16:04:17.343 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T16:04:17.343 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T16:04:17.343 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T16:04:17.343 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.379 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T16:04:17.379 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T16:04:17.379 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.453 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T16:04:17.454 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-17T16:04:17.454 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.523 INFO:tasks.ceph.osd.8:Stopped 2024-09-17T16:04:17.523 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-17T16:04:17.523 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.596 INFO:tasks.ceph.osd.12:Stopped 2024-09-17T16:04:17.596 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T16:04:17.596 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.633 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T16:04:17.634 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T16:04:17.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.704 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T16:04:17.704 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-17T16:04:17.705 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.773 INFO:tasks.ceph.osd.9:Stopped 2024-09-17T16:04:17.773 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-17T16:04:17.773 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.843 INFO:tasks.ceph.osd.13:Stopped 2024-09-17T16:04:17.843 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T16:04:17.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.873 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T16:04:17.873 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-17T16:04:17.873 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:17.940 INFO:tasks.ceph.osd.6:Stopped 2024-09-17T16:04:17.941 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-17T16:04:17.941 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.009 INFO:tasks.ceph.osd.10:Stopped 2024-09-17T16:04:18.009 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-17T16:04:18.009 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.083 INFO:tasks.ceph.osd.14:Stopped 2024-09-17T16:04:18.083 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T16:04:18.083 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.116 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T16:04:18.116 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-17T16:04:18.116 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.186 INFO:tasks.ceph.osd.7:Stopped 2024-09-17T16:04:18.186 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-17T16:04:18.187 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.263 INFO:tasks.ceph.osd.11:Stopped 2024-09-17T16:04:18.263 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-17T16:04:18.263 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.333 INFO:tasks.ceph.osd.15:Stopped 2024-09-17T16:04:18.333 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T16:04:18.334 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-17T16:04:18.334 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.365 INFO:tasks.ceph.mgr.y:Stopped 2024-09-17T16:04:18.365 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T16:04:18.365 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.406 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T16:04:18.406 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T16:04:18.406 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T16:04:18.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.421 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T16:04:18.421 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T16:04:18.421 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.435 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T16:04:18.436 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T16:04:18.436 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T16:04:18.450 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T16:04:18.451 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T16:04:18.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-17T16:04:18.501 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T16:04:18.502 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T16:04:18.746 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T16:04:18.746 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T16:04:18.955 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T16:04:18.955 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-17T16:04:19.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T16:04:19.021 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-17T16:04:19.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:04:19.079 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T16:04:19.212 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:04:19.213 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T16:04:19.292 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:04:19.292 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-17T16:04:19.350 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:04:19.350 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-17T16:04:19.409 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:04:19.409 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T16:04:19.723 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:04:19.723 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-17T16:04:19.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:04:19.796 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-17T16:04:19.852 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:04:19.852 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-17T16:04:19.910 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:04:19.910 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T16:04:20.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:04:20.237 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-17T16:04:20.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:04:20.317 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-17T16:04:20.392 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:04:20.392 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-17T16:04:20.448 INFO:tasks.ceph:Archiving mon data... 2024-09-17T16:04:20.449 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/data/mon.a.tgz 2024-09-17T16:04:20.449 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-09-17T16:04:20.469 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.tGpvKTUS1f 2024-09-17T16:04:20.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.tGpvKTUS1f 2024-09-17T16:04:20.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.tGpvKTUS1f 2024-09-17T16:04:20.742 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.tGpvKTUS1f is 479KB 2024-09-17T16:04:20.844 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.tGpvKTUS1f 2024-09-17T16:04:20.860 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/data/mon.b.tgz 2024-09-17T16:04:20.860 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-17T16:04:20.880 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.T4V6Mmyzq5 2024-09-17T16:04:20.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.T4V6Mmyzq5 2024-09-17T16:04:21.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.T4V6Mmyzq5 2024-09-17T16:04:21.189 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.T4V6Mmyzq5 is 684KB 2024-09-17T16:04:21.292 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.T4V6Mmyzq5 2024-09-17T16:04:21.308 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/data/mon.c.tgz 2024-09-17T16:04:21.308 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-09-17T16:04:21.327 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.Sa3HCepO7T 2024-09-17T16:04:21.327 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Sa3HCepO7T 2024-09-17T16:04:21.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.Sa3HCepO7T 2024-09-17T16:04:21.727 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.Sa3HCepO7T is 683KB 2024-09-17T16:04:21.828 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.Sa3HCepO7T 2024-09-17T16:04:21.842 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T16:04:21.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T16:04:21.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T16:04:21.847 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T16:04:21.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T16:04:21.933 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T16:04:21.951 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T16:04:21.966 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T16:04:21.985 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T16:04:21.985 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T16:04:21.987 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi006/crash 2024-09-17T16:04:21.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T16:04:22.018 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi050/crash 2024-09-17T16:04:22.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T16:04:22.048 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi169/crash 2024-09-17T16:04:22.049 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T16:04:22.075 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi181/crash 2024-09-17T16:04:22.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T16:04:22.116 INFO:tasks.ceph:Compressing logs... 2024-09-17T16:04:22.117 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:04:22.120 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:04:22.122 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:04:22.125 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T16:04:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-17T16:04:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T16:04:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T16:04:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T16:04:22.147 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-17T16:04:22.148 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T16:04:22.149 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-17T16:04:22.149 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T16:04:22.149 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log/var/log/ceph/ceph-osd.1.log: 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.13.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T16:04:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T16:04:22.155 INFO:teuthology.orchestra.run.smithi169.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T16:04:22.155 INFO:teuthology.orchestra.run.smithi050.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T16:04:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-09-17T16:04:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52777.log 2024-09-17T16:04:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52147.log 2024-09-17T16:04:22.189 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-09-17T16:04:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50994.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52777.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37781.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52147.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47201.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50994.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47862.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37781.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47201.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49711.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37746.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47862.log.gz 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52441.log 2024-09-17T16:04:22.190 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49711.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40036.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37746.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37771.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52441.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40036.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50425.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44379.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37771.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49837.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50425.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47509.log 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44379.log.gz 2024-09-17T16:04:22.191 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49837.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37761.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42109.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47509.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50677.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr: 28.4% -- replaced with /var/log/ceph/ceph-client.admin.37761.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.42109.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50677.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42541.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37763.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2024-09-17T16:04:22.192 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42541.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37763.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51017.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47447.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37759.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51017.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47447.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51490.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37759.log.gz 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-09-17T16:04:22.193 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47736.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51490.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39974.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47798.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47736.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38412.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39974.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37778.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.47798.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38412.log.gz 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-09-17T16:04:22.194 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37778.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49375.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47325.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49375.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47325.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-09-17T16:04:22.195 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51555.log 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38871.log 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51555.log.gz 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45627.log 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53113.log 2024-09-17T16:04:22.196 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38871.log.gz 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52273.log 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.45627.log.gz 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46219.log 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53113.log.gz 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52273.log.gz 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45514.log 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51620.log 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46219.log.gz 2024-09-17T16:04:22.197 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44357.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45514.log: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.45514.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49202.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51620.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49244.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.44357.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47571.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49202.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50257.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49244.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47571.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51651.log 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50257.log.gz 2024-09-17T16:04:22.198 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51003.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37760.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51651.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42894.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51003.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41720.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37760.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49753.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42894.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41720.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41720.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45189.log 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49753.log.gz 2024-09-17T16:04:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46685.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51636.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45189.log.gz 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46685.log.gz 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51636.log.gz 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44151.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43627.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40674.log 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-09-17T16:04:22.200 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43896.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43627.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.40674.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43896.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37783.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37783.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51519.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2024-09-17T16:04:22.201 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51519.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37445.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37445.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49963.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43189.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49459.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43189.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43189.log.gz 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49417.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38398.log 2024-09-17T16:04:22.202 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49459.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49417.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46177.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38398.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52357.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46177.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46177.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.51653.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48722.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52357.log.gz 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51653.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.51653.log.gz /var/log/ceph/ceph-client.admin.49501.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2024-09-17T16:04:22.203 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48722.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33768.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41882.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50990.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33768.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52525.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41882.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50990.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50299.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50990.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52525.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.52525.log.gz 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.52231.log 2024-09-17T16:04:22.204 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50299.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50991.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52231.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52945.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50991.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52945.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.40414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-09-17T16:04:22.205 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41802.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53029.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48681.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41802.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53029.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53029.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46798.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38912.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: 89.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48681.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33367.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46798.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38912.log.gz 2024-09-17T16:04:22.206 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33367.log.gz 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48207.log 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38385.log 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43554.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48207.log.gz 2024-09-17T16:04:22.207 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46997.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32973.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32773.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32973.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32973.log.gz 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32853.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32773.log.gz 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.37946.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32693.log 2024-09-17T16:04:22.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32853.log.gz 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/objectstore_tool.37946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36759.log 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32733.log 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32693.log.gz 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.0.36759.log: gzip -5 --verbose -- 75.4% /var/log/ceph/ceph-client.admin.32646.log 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.0.36759.log.gz 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33173.log 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32733.log.gz 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32646.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32646.log.gz 2024-09-17T16:04:22.227 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33173.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39130.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33173.log.gz 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33053.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/objectstore_tool.39130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33013.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33133.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33053.log.gz 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32893.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33013.log.gz 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33133.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38370.log 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33133.log.gz 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32893.log.gz 2024-09-17T16:04:22.228 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32813.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/objectstore_tool.38370.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32813.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32813.log.gz 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32562.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:gzip 93.7% -- replaced with /var/log/ceph/objectstore_tool.37946.log.gz 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32602.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32562.log: /var/log/ceph/ceph-mgr.x.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32562.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32933.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33093.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32602.log.gz 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.32933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32933.log.gz 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38158.log 2024-09-17T16:04:22.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33093.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T16:04:22.230 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33093.log.gz 2024-09-17T16:04:22.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/objectstore_tool.38158.log: 93.7% -- replaced with /var/log/ceph/objectstore_tool.38158.log.gz 2024-09-17T16:04:22.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.3.log: 93.3% -- replaced with /var/log/ceph/objectstore_tool.39130.log.gz 2024-09-17T16:04:22.232 INFO:teuthology.orchestra.run.smithi181.stderr: 93.9% -- replaced with /var/log/ceph/objectstore_tool.38370.log.gz 2024-09-17T16:04:22.257 INFO:teuthology.orchestra.run.smithi181.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38385.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44109.log 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46997.log.gz 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37228.log 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37361.log 2024-09-17T16:04:22.293 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44109.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37228.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50635.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37228.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46882.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37361.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52819.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50635.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53449.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr: 25.4% -- replaced with /var/log/ceph/ceph-client.admin.46882.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45781.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52819.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52609.log 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53449.log.gz 2024-09-17T16:04:22.294 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45781.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39167.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52609.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49291.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49669.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39167.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38203.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49291.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53323.log 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49669.log.gz 2024-09-17T16:04:22.295 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46392.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38203.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51014.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46392.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.46392.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49543.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53741.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51014.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49543.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.53533.log -- replaced with /var/log/ceph/ceph-client.admin.49543.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53741.log.gz 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51505.log 2024-09-17T16:04:22.296 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53533.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53071.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45077.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50983.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53071.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45077.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45077.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.37780.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50983.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53281.log 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-09-17T16:04:22.297 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38999.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.50972.log -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45718.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53281.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39209.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50972.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45718.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39209.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49585.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52735.log 2024-09-17T16:04:22.298 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49585.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39796.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51648.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52735.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39796.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39796.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.47074.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53701.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51648.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47074.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47074.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.51001.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53701.log.gz 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51001.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51001.log.gz -5 2024-09-17T16:04:22.299 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42935.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41944.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39856.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.42935.log.gz 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41944.log.gz 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38402.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.39856.log.gz 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38402.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.38402.log.gz /var/log/ceph/ceph-client.admin.40547.log 2024-09-17T16:04:22.300 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37767.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51018.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50509.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37767.log.gz 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51634.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51018.log.gz 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42225.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50509.log.gz 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51634.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.52189.log 2024-09-17T16:04:22.301 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51634.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50131.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42225.log: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.42225.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42684.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52189.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50131.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.47263.log -- replaced with /var/log/ceph/ceph-client.admin.50131.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42684.log: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.42684.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46840.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51016.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-09-17T16:04:22.302 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46840.log.gz 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46290.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51016.log.gz 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52315.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46290.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46290.log.gz 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52315.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.52315.log.gz /var/log/ceph/ceph-client.admin.48916.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50551.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48916.log.gz 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49921.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40232.log 2024-09-17T16:04:22.303 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50551.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49921.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38418.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40232.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47384.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52903.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.38418.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43362.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47384.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38308.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52903.log.gz 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43362.log: /var/log/ceph/ceph-client.admin.41572.log 2024-09-17T16:04:22.304 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43362.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38308.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47975.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45034.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51569.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.47975.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37147.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45034.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.51569.log: -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51569.log.gz 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.39775.log 2024-09-17T16:04:22.305 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37147.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53659.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44024.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37787.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53659.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44024.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44024.log.gz -5 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41419.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48147.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37787.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51650.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41419.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50173.log 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48147.log.gz 2024-09-17T16:04:22.306 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.51650.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51650.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50173.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48540.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38400.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.51393.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40455.log: 25.8% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44964.log 2024-09-17T16:04:22.307 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51749.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44964.log: gzip -5 --verbose 24.2% -- /var/log/ceph/ceph-client.admin.49795.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44964.log.gz 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51749.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.51749.log.gz --verbose 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.49795.log: /var/log/ceph/ceph-client.admin.38364.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49795.log.gz/var/log/ceph/ceph.log: 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53239.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38364.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37765.log 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38364.log.gz 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53239.log.gz 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-17T16:04:22.308 INFO:teuthology.orchestra.run.smithi006.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37743.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49118.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37765.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48597.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37743.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44067.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49118.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52105.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48597.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49627.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44067.log.gz 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52105.log: -- /var/log/ceph/ceph-client.admin.40535.log 2024-09-17T16:04:22.309 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52105.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49627.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50005.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37769.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50005.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50005.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50341.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49879.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37769.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47673.log 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50341.log.gz 2024-09-17T16:04:22.310 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49879.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51002.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49879.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38350.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47673.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52861.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51002.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38350.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38257.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38350.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49160.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52861.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38257.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49160.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45126.log 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49160.log.gz 2024-09-17T16:04:22.311 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45585.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33649.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45126.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45585.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50999.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45585.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50975.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.33649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33649.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47591.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50999.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50975.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.47934.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50975.log.gz 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46372.log 2024-09-17T16:04:22.312 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47591.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47934.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46372.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43982.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51019.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50047.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51019.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51019.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43828.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50047.log.gz 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44534.log 2024-09-17T16:04:22.313 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43828.log: gzip 24.5% -- replaced with /var/log/ceph/ceph-client.admin.43828.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51618.log 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44534.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44534.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51618.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50383.log 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.53155.log: -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.39393.log 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50383.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52567.log 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.39393.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42183.log 25.3% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52567.log.gz 2024-09-17T16:04:22.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44678.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.42183.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44454.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42183.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48269.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44678.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44454.log: /var/log/ceph/ceph-client.admin.49333.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0%/var/log/ceph/ceph-client.admin.48269.log: -- replaced with /var/log/ceph/ceph-client.admin.44454.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 24.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37403.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48269.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.49333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39041.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49333.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37403.log: -- /var/log/ceph/ceph-client.admin.43290.log 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37403.log.gz 2024-09-17T16:04:22.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39041.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46902.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43290.log: gzip 24.5% -- replaced with /var/log/ceph/ceph-client.admin.43290.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46902.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46902.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37786.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.52063.log: --verbose -- /var/log/ceph/ceph-client.admin.44317.log 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37786.log.gz 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27518.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.44317.log: -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.43402.log 2024-09-17T16:04:22.316 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44317.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph.tmp-client.admin.27518.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27518.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44760.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.43402.log: gzip -5 --verbose 24.5% -- replaced with /var/log/ceph/ceph-client.admin.43402.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.53575.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44760.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50089.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44235.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5/var/log/ceph/ceph-client.admin.50089.log: --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50089.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr: /var/log/ceph/ceph-client.admin.52651.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44235.log.gz 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52483.log 2024-09-17T16:04:22.317 INFO:teuthology.orchestra.run.smithi006.stderr:gzip/var/log/ceph/ceph-client.admin.52651.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52483.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47139.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.39125.log: gzip/var/log/ceph/ceph-client.admin.41439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.38399.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47139.log: /var/log/ceph/ceph-client.admin.38399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46727.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47139.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38399.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.40854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47778.log 2024-09-17T16:04:22.318 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.46727.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.46727.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42821.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.47778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47778.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53407.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42821.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.42821.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38406.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53407.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42440.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.38406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38406.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44275.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42440.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53491.log 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44275.log.gz 2024-09-17T16:04:22.319 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48639.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53491.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53491.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37319.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48639.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.37319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37319.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51747.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.41332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.51747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51747.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.45698.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr: -- /var/log/ceph/ceph-client.admin.44658.log 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-09-17T16:04:22.320 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52693.log 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.44658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44658.log.gz 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53365.log 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.52693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52693.log.gz 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50467.log 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.53365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53365.log.gz 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.50467.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50467.log.gz 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48484.log 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.42028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-09-17T16:04:22.321 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/ceph-client.admin.48484.log: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.48484.log.gz 2024-09-17T16:04:22.322 INFO:teuthology.orchestra.run.smithi169.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T16:04:22.332 INFO:teuthology.orchestra.run.smithi050.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T16:04:22.484 INFO:teuthology.orchestra.run.smithi181.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T16:04:22.490 INFO:teuthology.orchestra.run.smithi006.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T16:04:22.496 INFO:teuthology.orchestra.run.smithi050.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-17T16:04:22.561 INFO:teuthology.orchestra.run.smithi050.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T16:04:22.632 INFO:teuthology.orchestra.run.smithi169.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-17T16:04:22.662 INFO:teuthology.orchestra.run.smithi169.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T16:04:22.677 INFO:teuthology.orchestra.run.smithi181.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-17T16:04:22.703 INFO:teuthology.orchestra.run.smithi169.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-17T16:04:22.839 INFO:teuthology.orchestra.run.smithi006.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-17T16:04:22.878 INFO:teuthology.orchestra.run.smithi169.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-17T16:04:22.880 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-09-17T16:04:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m0.743s 2024-09-17T16:04:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m2.369s 2024-09-17T16:04:22.880 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.093s 2024-09-17T16:04:22.885 INFO:teuthology.orchestra.run.smithi181.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-17T16:04:22.963 INFO:teuthology.orchestra.run.smithi181.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-17T16:04:22.965 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-17T16:04:22.965 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m0.788s 2024-09-17T16:04:22.965 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m2.270s 2024-09-17T16:04:22.965 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.122s 2024-09-17T16:04:23.102 INFO:teuthology.orchestra.run.smithi050.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T16:04:23.130 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T16:04:23.230 INFO:teuthology.orchestra.run.smithi006.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-17T16:04:23.230 INFO:teuthology.orchestra.run.smithi006.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T16:04:23.232 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-09-17T16:04:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m1.097s 2024-09-17T16:04:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m3.950s 2024-09-17T16:04:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.304s 2024-09-17T16:04:23.449 INFO:teuthology.orchestra.run.smithi050.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-17T16:04:23.451 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-09-17T16:04:23.451 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m1.313s 2024-09-17T16:04:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m3.110s 2024-09-17T16:04:23.452 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m0.107s 2024-09-17T16:04:23.452 INFO:tasks.ceph:Archiving logs... 2024-09-17T16:04:23.452 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi006/log 2024-09-17T16:04:23.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T16:04:24.312 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi050/log 2024-09-17T16:04:24.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T16:04:24.819 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi169/log 2024-09-17T16:04:24.820 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T16:04:25.226 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi181/log 2024-09-17T16:04:25.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T16:04:25.640 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T16:04:25.650 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T16:04:25.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T16:04:25.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T16:04:25.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T16:04:25.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T16:04:25.777 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T16:04:25.787 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T16:04:25.787 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T16:04:25.790 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T16:04:25.793 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T16:04:25.796 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T16:04:25.805 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-09-17T16:04:25.808 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-17T16:04:25.811 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-09-17T16:04:25.812 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T16:04:25.813 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-09-17T16:04:25.813 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -81us[ -75us] +/- 44ms 2024-09-17T16:04:25.813 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 +455us[ +461us] +/- 50ms 2024-09-17T16:04:25.813 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -458us[ -452us] +/- 70ms 2024-09-17T16:04:25.813 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:04:25.814 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-17T16:04:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T16:04:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-17T16:04:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -71us[ -66us] +/- 44ms 2024-09-17T16:04:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 +458us[ +458us] +/- 50ms 2024-09-17T16:04:25.816 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -448us[ -443us] +/- 70ms 2024-09-17T16:04:25.817 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:04:25.818 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T16:04:25.818 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-09-17T16:04:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -62us[ -48us] +/- 44ms 2024-09-17T16:04:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 +481us[ +481us] +/- 50ms 2024-09-17T16:04:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -426us[ -426us] +/- 70ms 2024-09-17T16:04:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:04:25.821 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T16:04:25.821 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-17T16:04:25.821 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 -69us[ -63us] +/- 44ms 2024-09-17T16:04:25.821 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 +462us[ +468us] +/- 50ms 2024-09-17T16:04:25.822 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -447us[ -447us] +/- 70ms 2024-09-17T16:04:25.822 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T16:04:25.822 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T16:04:25.837 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T16:04:25.838 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T16:04:25.853 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T16:04:25.933 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T16:04:26.008 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T16:04:26.080 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T16:04:26.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T16:04:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:26.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T16:04:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:26.217 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T16:04:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:26.248 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T16:04:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:26.278 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T16:04:26.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T16:04:26.300 INFO:teuthology.task.internal:Duration was 655.486835 seconds 2024-09-17T16:04:26.301 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T16:04:26.310 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T16:04:26.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T16:04:26.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T16:04:26.316 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T16:04:26.318 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T16:04:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T16:04:26.358 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T16:04:26.358 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T16:04:26.359 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T16:04:26.814 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T16:04:26.815 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-09-17T16:04:26.815 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T16:04:26.840 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-09-17T16:04:26.840 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T16:04:26.871 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-09-17T16:04:26.871 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T16:04:26.895 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T16:04:26.896 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T16:04:26.928 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T16:04:26.928 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T16:04:26.930 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T16:04:26.933 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T16:04:26.938 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T16:04:26.997 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T16:04:26.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T16:04:27.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T16:04:27.006 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T16:04:27.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T16:04:27.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T16:04:27.155 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T16:04:27.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T16:04:27.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T16:04:27.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T16:04:27.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T16:04:27.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T16:04:27.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.275 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.294 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-09-17T16:04:27.296 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-09-17T16:04:27.298 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-09-17T16:04:27.320 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-17T16:04:27.342 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:27.373 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:27.388 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:27.403 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T16:04:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T16:04:27.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T16:04:27.428 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T16:04:27.429 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi006 2024-09-17T16:04:27.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T16:04:27.466 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi050 2024-09-17T16:04:27.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T16:04:27.501 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi169 2024-09-17T16:04:27.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T16:04:27.534 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909015/remote/smithi181 2024-09-17T16:04:27.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T16:04:27.568 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T16:04:27.568 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T16:04:27.570 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T16:04:27.573 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T16:04:27.575 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T16:04:27.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T16:04:28.209 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T16:04:28.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T16:04:28.224 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T16:04:28.224 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T16:04:28.226 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T16:04:28.229 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T16:04:28.231 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T16:04:28.241 INFO:teuthology.orchestra.run.smithi006.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 16:04 /home/ubuntu/cephtest 2024-09-17T16:04:28.245 INFO:teuthology.orchestra.run.smithi169.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 16:04 /home/ubuntu/cephtest 2024-09-17T16:04:28.245 INFO:teuthology.orchestra.run.smithi050.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 16:04 /home/ubuntu/cephtest 2024-09-17T16:04:28.247 INFO:teuthology.orchestra.run.smithi181.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 16:04 /home/ubuntu/cephtest 2024-09-17T16:04:28.248 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T16:04:28.258 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T16:04:28.281 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 655.4868354797363 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T16:04:28.282 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T16:04:28.438 INFO:teuthology.run:pass